Skip to main content
Best Web Developer logo
Language and currency EN £ GBP

Language

    Currency

    Prices on this site are shown in the currency you choose.

    SEO Tools

    Robots.txt Generator

    A small file that is easy to get catastrophically wrong, so it is worth generating rather than typing.

    • Runs in your browser
    • Nothing uploaded

    The tool

    Add the paths you want to keep crawlers out of, and your sitemap URL.

    Everything you enter stays on your own machine. Nothing is sent to us or to anyone else, which you can confirm in your browser network tab.

    What people use it for

    • Keeping crawlers out of admin, cart and search result pages
    • Pointing search engines at your sitemap
    • Blocking an entire staging site from being indexed
    • Reviewing an existing robots.txt for obvious mistakes
    • Producing a starting file for a new site

    About this tool

    The one rule to be careful with

    A robots.txt containing Disallow: / asks every crawler to ignore the entire site. It is a single character away from a file that blocks nothing, and it has removed more sites from search results than almost any other single mistake.

    The option exists here because staging sites genuinely need it, and the label says so plainly. Check the generated file before you upload it.

    What robots.txt does not do

    It requests that crawlers stay out. It does not keep a page out of the index when other sites link to it, and it offers no security: the file is public, so listing a sensitive path advertises it. Use noindex for keeping pages out of results, and authentication for keeping people out.

    Check it after uploading

    Fetch your own robots.txt in a browser after deploying it. Crawlers treat a 404, or the wrong content type, as permission to crawl everything, which may or may not match your intention.

    Search Console and equivalent tools also show how they read the file, which is worth checking after any change.

    Need this handled across a whole website? See our Technical SEO Services page.

    Questions

    Questions about this tool

    The things clients actually want to know — costs, timelines, ownership and what happens after launch.

    In the root of your domain, reachable at /robots.txt. It will not work anywhere else.

    No. Use a noindex tag for that. Robots.txt asks crawlers not to visit, which is different from asking search engines not to list.

    No, and treating it as one is a mistake. The file is public, so listing a sensitive path tells everyone it exists.

    Usually yes, to save crawl budget. But protect it with authentication rather than relying on this file.

    Reputable ones do. Malicious ones ignore it entirely, which is another reason it is not a security control.

    Free strategy session

    Let’s build something worth finding.

    Tell us where you want the business to be in twelve months. We will come back with a costed plan, a realistic timeline, and the numbers we expect to move — no obligation, no jargon.

    • 30-minute call, no pitch deck
    • Fixed-price proposal in 48 hours
    • You own everything we build

    Talk to a developer

    Not a sales team

    Response time
    Within 1 business hour
    Where we work
    UK, USA & the Gulf
    Contracts
    No lock-in, monthly rolling
    Working since
    2023
    infobestwebdeveloper.org

    Free strategy session

    Let’s build something worth finding.

    Tell us where you want the business to be in twelve months. We will come back with a costed plan, a realistic timeline, and the numbers we expect to move — no obligation, no jargon.

    • 30-minute call, no pitch deck
    • Fixed-price proposal in 48 hours
    • You own everything we build

    Talk to a developer

    Not a sales team

    Response time
    Within 1 business hour
    Where we work
    UK, USA & the Gulf
    Contracts
    No lock-in, monthly rolling
    Working since
    2023
    infobestwebdeveloper.org
    Free consultation

    show prices in SGD?

    Video