SEO Tools
Robots.txt Generator
A small file that is easy to get catastrophically wrong, so it is worth generating rather than typing.
- Runs in your browser
- Nothing uploaded
The tool
Add the paths you want to keep crawlers out of, and your sitemap URL.
Everything you enter stays on your own machine. Nothing is sent to us or to anyone else, which you can confirm in your browser network tab.
What people use it for
- Keeping crawlers out of admin, cart and search result pages
- Pointing search engines at your sitemap
- Blocking an entire staging site from being indexed
- Reviewing an existing robots.txt for obvious mistakes
- Producing a starting file for a new site
About this tool
The one rule to be careful with
A robots.txt containing Disallow: / asks every crawler to ignore the entire site. It is a single character away from a file that blocks nothing, and it has removed more sites from search results than almost any other single mistake.
The option exists here because staging sites genuinely need it, and the label says so plainly. Check the generated file before you upload it.
What robots.txt does not do
It requests that crawlers stay out. It does not keep a page out of the index when other sites link to it, and it offers no security: the file is public, so listing a sensitive path advertises it. Use noindex for keeping pages out of results, and authentication for keeping people out.
Check it after uploading
Fetch your own robots.txt in a browser after deploying it. Crawlers treat a 404, or the wrong content type, as permission to crawl everything, which may or may not match your intention.
Search Console and equivalent tools also show how they read the file, which is worth checking after any change.
Need this handled across a whole website? See our Technical SEO Services page.
More tools and services
- Meta Length Checker for Titles and Descriptions
- Keyword Density Checker
- Meta Tag Generator
- Schema Markup Generator
- Hreflang Tag Generator
If you would rather someone did this properly for you, we offer Web Design Services and Website Development, SEO Services: Search Engine Optimization, AI Automation Agency.
Questions
Questions about this tool
The things clients actually want to know — costs, timelines, ownership and what happens after launch.
In the root of your domain, reachable at /robots.txt. It will not work anywhere else.
No. Use a noindex tag for that. Robots.txt asks crawlers not to visit, which is different from asking search engines not to list.
No, and treating it as one is a mistake. The file is public, so listing a sensitive path tells everyone it exists.
Usually yes, to save crawl budget. But protect it with authentication rather than relying on this file.
Reputable ones do. Malicious ones ignore it entirely, which is another reason it is not a security control.
Free strategy session
Let’s build something worth finding.
Tell us where you want the business to be in twelve months. We will come back with a costed plan, a realistic timeline, and the numbers we expect to move — no obligation, no jargon.
- 30-minute call, no pitch deck
- Fixed-price proposal in 48 hours
- You own everything we build
Talk to a developer
Not a sales team
- Response time
- Within 1 business hour
- Where we work
- UK, USA & the Gulf
- Contracts
- No lock-in, monthly rolling
- Working since
- 2023