Crawling
AI Bot Access Tester
Separate the three questions people mix up: what robots.txt permits, what the server returns, and what the page says about indexing.
- Fetched live by our server
- Nothing stored
The tool
List up to twenty URLs, choose the crawlers, and we test each pair.
Loading the tool…
The URL you enter is sent to our server, which requests that page from the public internet and returns the response to your browser for analysis. We do not store the URL or the response, and we do not add either to analytics. Requests come from our server, not from your device or from the crawlers being simulated, so a site that blocks data-centre addresses or verifies crawlers by IP may answer differently.
What this tool checks
- Fetches robots.txt once per host and applies the group each crawler would follow, with the deciding line
- Requests each URL with the crawler's published user agent and records the status code, redirects and final URL
- Reads robots meta tags and X-Robots-Tag headers, including directives scoped to a specific crawler
- Explains what each crawler is for: search indexing, model training or fetching a page because a user asked
- Notes which tokens, such as Google-Extended, are robots.txt controls only and never fetch pages themselves
- Exports every result with its evidence and the test time as CSV
How to read the results
- Read the robots.txt column as permission only. Blocked means the operator says that crawler will not fetch the URL, if it follows its own documentation.
- Read the HTTP column as what our server received when it sent that crawler's user agent. A 403 or 429 there often means a firewall or bot-protection rule.
- Read the indexing column for noindex, nofollow or none. Those apply to search engines that honour them and say nothing about AI training.
- Check the registry table before acting: blocking a user-triggered agent such as ChatGPT-User affects people asking about your page, not training.
About this tool
Three different questions
When people ask whether an AI crawler can access their site, they usually mean three separate things, and mixing them up leads to the wrong fix.
- Is it permitted? robots.txt states which URLs a named crawler may request. Operators that follow the Robots Exclusion Protocol check it before fetching.
- Is it reachable? Your server, CDN or firewall decides what a request actually gets back. Many bot-protection services answer unknown or unverified crawlers with a 403, a 429 or a challenge page, whatever robots.txt says.
- Is it indexable? Robots meta tags and the X-Robots-Tag header tell search engines whether to index a page they have fetched. They are not a standard way to opt out of AI training.
This tool answers each question in its own column, so you can see, for example, that GPTBot is permitted by robots.txt but refused by your firewall.
Crawlers by purpose
AI companies now publish several user agents with different jobs. Treat them separately:
- Search crawlers such as Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot build an index that answers and search results are drawn from. Blocking them generally removes you from that product.
- Training crawlers such as GPTBot, ClaudeBot, CCBot and Meta-ExternalAgent collect content that may be used to train models. Google-Extended and Applebot-Extended are tokens only: they never fetch pages, and you use them in robots.txt to control whether content already fetched by Googlebot or Applebot may be used for AI models.
- User-triggered agents such as ChatGPT-User, Claude-User and Perplexity-User fetch a page because a person asked about it. Operators describe these as acting for the user, and some say robots.txt may not apply to them in every case.
Writing rules for each crawler
Name each crawler in its own group. A crawler that finds a group with its name ignores the * group, so rules you want everyone to follow must be repeated.
# Allow AI search, block training
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpWhen the HTTP column disagrees with robots.txt
If robots.txt allows a crawler but the request returns 403, 429 or 503, look at your CDN or security plugin. Many offer a setting for verified bots, which checks the crawler’s published IP ranges. Our test sends only the user agent string from our own server, so a verified-bot rule can block our request and still let the real crawler through. A failure here is a prompt to check those settings, not proof that the real crawler is blocked.
What no tool can tell you
There is no public way to confirm whether a specific page has been used for training or will be quoted in an AI answer. What you can control is permission, reachability and clear, accurate content. Be wary of any service that promises AI citations.
Limits and honest caveats
- Requests come from our server with the crawler's user agent string, not from the operator's own IP addresses. Firewalls that verify crawlers by reverse DNS or published IP ranges will treat our request differently from the real crawler.
- Nothing here can show whether a page has been indexed, used for training or cited in an AI answer. Those decisions happen inside each operator's systems.
- Some user-triggered agents are documented as not following robots.txt in every case, so permission and behaviour can differ.
- Pages that build their content with JavaScript are analysed as raw HTML here. Use Fetch and Render to compare.
- The crawler list is updated by hand from each operator's documentation and shows the date it was last reviewed.
What people use it for
- Checking whether a new robots.txt blocks AI training crawlers but still allows AI search
- Finding out why an AI assistant says it cannot open your page
- Confirming that a firewall or bot-protection rule is not refusing Googlebot or Bingbot
- Documenting the current AI crawler policy of a site for a client or legal team
- Comparing several sections of a site after a CMS or CDN change
Documentation and references
- robots.txt: how it works and how to get it right
- OpenAI: Overview of OpenAI crawlers (opens in a new tab)
- Anthropic: Does Anthropic crawl data from the web? (opens in a new tab)
- Google: Common crawlers, including Google-Extended (opens in a new tab)
- Perplexity: Perplexity crawlers (opens in a new tab)
- RFC 9309: Robots Exclusion Protocol (opens in a new tab)
Questions
Questions about this tool
The things clients actually want to know — costs, timelines, ownership and what happens after launch.
Not necessarily. GPTBot is OpenAI's training crawler. OAI-SearchBot powers search features and ChatGPT-User fetches pages on request, so each needs its own rule.
A robots.txt token, not a crawler. Disallowing it tells Google not to use content fetched by its crawlers for Gemini models and grounding. It does not affect Google Search.
Your server or CDN refused the request. Bot protection often blocks requests that claim to be a crawler but come from unverified addresses, which includes our test server.
Not in general. noindex is an instruction to search engines about indexing. Use robots.txt rules for the specific training crawler instead.
It is reviewed by hand against each operator's documentation. The review date is shown in the results.
Documentation last reviewed: by the Best Web Developer engineering team.
Free strategy session
Let’s build something worth finding.
Tell us where you want the business to be in twelve months. We will come back with a costed plan, a realistic timeline, and the numbers we expect to move — no obligation, no jargon.
- 30-minute call, no pitch deck
- Fixed-price proposal in 48 hours
- You own everything we build
Talk to a developer
Not a sales team
- Response time
- Within 1 business hour
- Where we work
- UK, USA & the Gulf
- Contracts
- No lock-in, monthly rolling
- Working since
- 2023