The short version
robots.txt is a plain text file at the root of a host that tells crawlers which URLs they may request. It controls crawling, not indexing, and it is not access control. Use it to keep crawlers out of areas that waste crawl effort, such as internal search results or endless filter combinations. Do not use it to hide pages from search results or to protect private content.
Where the file lives
The file must be at /robots.txt on each host and protocol it applies to. https://example.com/robots.txt covers only https://example.com. The www subdomain, other subdomains and the http version each need their own file, or a redirect to one. It must be served as plain text, encoded as UTF-8, and Google reads only the first 500 KiB.
Syntax
The file is made of groups. Each group starts with one or more User-agent lines naming crawlers, followed by Allow and Disallow rules. Sitemap lines can appear anywhere and apply to the whole file. Everything after # is a comment.
# Default rules for every crawler without its own group
User-agent: *
Disallow: /search
Disallow: /cart/
Allow: /cart/help/
# Bing gets the same rules plus one more
User-agent: Bingbot
Disallow: /search
Disallow: /cart/
Allow: /cart/help/
Disallow: /beta/
Sitemap: https://example.com/sitemap.xmlField names are not case-sensitive. Paths are case-sensitive and match from the start of the URL path, including the query string. Each rule should start with / or *.
Which group a crawler follows
A crawler looks for groups whose User-agent matches its product token, such as Googlebot or GPTBot, ignoring case. If it finds any, it combines their rules and ignores everything else, including the * group. Only a crawler with no group of its own falls back to *. That is why the Bingbot group above repeats the general rules: without them Bingbot would crawl /search.
Google adds one convenience: its specialised crawlers, such as Googlebot-Image and Googlebot-News, follow the Googlebot group when no group names them.
Which rule wins
Inside the chosen group, the rule with the longest matching path wins, whatever order the rules are in. If an Allow and a Disallow rule are equally long, Allow wins. Two wildcards are supported:
*matches any run of characters, including none.$at the end of a rule means the URL must end there.
User-agent: *
Disallow: /*.pdf$ # blocks /guide.pdf, not /guide.pdf?v=2
Disallow: /*?sessionid= # blocks any URL with that parameter
Allow: /downloads/brochure.pdf$The Allow rule is longer than /*.pdf$, so the brochure stays crawlable while other PDFs are blocked.
Status codes
- 200: the rules are used.
- 3xx: Google follows up to five redirects and uses the result.
- 4xx, including 404: treated as if there were no robots.txt, so everything may be crawled.
- 5xx or no response: Google stops crawling the site for a while, as if everything were disallowed, and uses its last cached copy if it has one. A long-running server error on robots.txt can therefore halt crawling of the whole site.
Google caches robots.txt for up to 24 hours, so changes are not instant.
AI crawlers
AI companies publish separate user agents for different purposes, and each needs its own rule. Training crawlers include GPTBot, ClaudeBot and CCBot. Search and answer crawlers include OAI-SearchBot, Claude-SearchBot and PerplexityBot. User-triggered agents such as ChatGPT-User fetch a page when someone asks about it. Google-Extended and Applebot-Extended are tokens only: they never fetch anything, and you use them to control whether content Google or Apple already crawls may be used for AI models, without affecting search.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /Decide crawler by crawler and record why. Blocking every AI agent also removes you from AI search features that could send visitors.
Common mistakes
- Blocking CSS, JavaScript or images. Search engines render pages. If they cannot load your assets, they may see a broken or empty page.
- Using robots.txt to remove pages from search. A blocked page can still be indexed from links, and crawlers cannot see a noindex tag on a page they may not fetch. Allow crawling and use
noindexinstead. Noindex:in robots.txt. Google stopped supporting it in 2019.- Leftover
Disallow: /from a staging site that went live. - Relying on
Crawl-delay. Google ignores it. Bing still reads it. - Listing secret paths. The file is public, so it advertises what it lists. Protect private areas with authentication.
Verifying a change
- Draft the new file and test it against a list of important URLs, including page templates and the assets they load, with the robots.txt Tester.
- Check AI and search crawlers separately with the AI Bot Access Tester.
- Publish the file, fetch
/robots.txtin a browser and confirm it returns 200 with the new content. - In Google Search Console, open the robots.txt report to see the version Google fetched, and request a recrawl if it is urgent.