robots.txt Tester

What robots.txt does, and what it does not

A robots.txt file sits at the root of a host, for example https://example.com/robots.txt, and tells well-behaved crawlers which URLs they may request. It is a crawling control, not an indexing control and not a security measure. A URL that is disallowed can still be indexed without its content if other pages link to it, and anyone can still open it in a browser. To keep a page out of search results, let it be crawled and use a noindex robots meta tag or an X-Robots-Tag header instead.

Each host and protocol has its own file. Rules on https://www.example.com/robots.txt do not apply to https://shop.example.com/ or to the http:// version of the same host.

How a crawler chooses which rules to follow

The file is divided into groups. A group starts with one or more User-agent lines and continues with Allow and Disallow rules until the next User-agent line. A crawler looks for the group whose user-agent matches its own product token, ignoring case. If several groups name the same crawler, their rules are combined. Only if no group names it does the crawler fall back to the * group. A crawler that finds its own group ignores the * group entirely, which surprises many site owners.

User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow: /tmp/

Here Googlebot may crawl /private/, because its own group does not mention it. Google also documents that its specialised crawlers, such as Googlebot-Image, follow the Googlebot group when no group names them directly.

Which rule wins

Within the applicable group, the rule with the longest matching path wins. If an Allow and a Disallow rule match with equal length, the Allow wins. Two special characters are supported: * matches any sequence of characters, and $ anchors the end of the URL. Matching is case-sensitive and includes the query string.

User-agent: *
Disallow: /shop/
Allow: /shop/products/
Disallow: /*?sort=
Disallow: /*.pdf$

With these rules /shop/products/chair is allowed because the Allow rule is longer, /shop/products/chair?sort=price is blocked because the query-string rule is longer still, and /guide.pdf?download=1 is allowed because $ requires the URL to end in .pdf.

Status codes matter as much as the rules

Google treats a robots.txt that returns a 4xx status, such as 404, as if there were no restrictions. A 5xx status or a timeout is different: Google stops crawling the site for a while, as if everything were disallowed, and falls back to a cached copy if it has one. That is why this tester reports such cases as Indeterminate rather than guessing. Redirects are followed for up to five hops.

Mistakes we see most often

• Blocking stylesheets, scripts or images that pages need to render. Search engines render pages much like a browser, and blocked resources can make a page look broken or empty to them.
• Leaving Disallow: / in place after a staging site goes live.
• Expecting Noindex: lines in robots.txt to work. Google stopped supporting them in 2019.
• Expecting Crawl-delay to slow Googlebot. Google ignores it; Bing still reads it.
• Blocking a page you also mark noindex. If crawlers cannot fetch the page, they never see the noindex tag.

Testing changes safely

Paste a draft into the tester with a list of your important URLs, including templates such as product, category and article pages and the CSS and JavaScript files they load. Fix anything that is blocked unexpectedly before you publish. After publishing, run the live test again, and use the robots.txt report in Google Search Console to confirm which version Google has fetched.