{"id":69602,"date":"2022-07-31T06:09:58","date_gmt":"2022-07-31T04:09:58","guid":{"rendered":"https:\/\/rmarketingdigital.com\/wiki\/robots-txt-2\/"},"modified":"2024-10-28T22:48:28","modified_gmt":"2024-10-28T21:48:28","slug":"robots-txt-2","status":"publish","type":"glossary","link":"https:\/\/rmarketingdigital.com\/en\/wiki\/robots-txt-2\/","title":{"rendered":"Robots.txt"},"content":{"rendered":"

<\/p>\n

\n

The robots.txt file<\/b> is a document that establishes which parts of a domain can be analyzed by search engine crawlers and provides a link to the XML-sitemap<\/i>.\n<\/p>\n

\nStructure<\/span>
\n<\/h2>\n

The call "Robots Exclusion Standard Protocol"<\/i>, Standard Protocol for Robots Exclusions<\/b>, was first published in 1994. This protocol defines that search engine crawlers must find and read the file named "Robots.txt"<\/b> before starting indexing. That is why it should be placed in the root directory of the domain. In spite of everything, we must remember that not all trackers follow this same rule and therefore, the "Robots.txt"<\/b> they do not promise the 100% access and privacy protection. Some search engines still index blocked pages and even show those with no description in the SERPs. This is particularly the case with websites that contain too many links. Regardless, the major search engines like Google, Yahoo<\/b> and Bing<\/b> yes they conform to the protocol rules "Robots.txt"<\/b>.\n<\/p>\n

\nCreation and control of the "robots.txt"<\/span>
\n<\/h2>\n

It is simple to create a "Robots.txt"<\/b> with the help of a text editor. At the same time, you can find free tools on the internet that offer detailed information about how to generate a file. "Robots.txt"<\/b> or that they even automatically create it for you. Each file contains 2 blocks. In the first, it is specified for which users the instructions are valid. In the second block the instructions are written, called "Disallow"<\/i>, with the list of pages to be excluded. It is recommended to check carefully that the file has been written correctly before downloading it to the directory since, with basically a tiny syntax error, you can misinterpret the instructions and index pages that, in theory, should not appear in the search results . To check if the file "Robots.txt"<\/b> works correctly you can use Google's webmaster tool and perform an analysis on \u201estatus\u201c -> \u201eblocked URLs\u201c.\n<\/p>\n

\"600x400-robotstxt-en-01.png\"<\/a>\n<\/p>\n

\nPages exclusion<\/span>
\n<\/h2>\n

The simplest structure of a file robots.txt<\/b> appears as follows:\n<\/p>\n

User-agent: Googlebot Disallow:\n<\/pre>\n
This code allows Googlebot to analyze all pages. The opposite, such as the complete ban of the web portal, is written as follows: '\n<\/p>\n
User-agent: Googlebot Disallow:\n<\/pre>\n
In the "User-agent" line the user writes who it is addressed to. The following terms can be used:\n<\/p>\n