Robots.txt Generator

Build a robots.txt file for your site. Allow everything, block everything, or set custom rules with disallow paths, a crawl delay and your sitemap URL, then copy a ready-to-upload file. Free and instant in your browser.

Read the guide: How to Write a robots.txt File
robots.txt
robots.txt
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Save this as robots.txt in your site root, served at /robots.txt.

How it works

  1. 1

    Pick a starting point

    Choose allow all to let crawlers in, block all to keep them out, or custom to set your own rules.

  2. 2

    Add your rules

    In custom mode, list the paths to disallow, set an optional crawl delay, and choose the user-agent the rules apply to.

  3. 3

    Add the sitemap and copy

    Point crawlers at your sitemap URL, then copy the file and upload it to your site root.

Instant & 100% private — nothing is uploaded

Everything runs locally in your browser. Your code, text and files are processed on your own device and are never sent to a server — so there are no upload waits, no size limits from us, and nothing is ever stored or logged.

Frequently asked questions

Where does robots.txt go?
It must sit at the root of your domain and be served at /robots.txt, for example https://example.com/robots.txt. Crawlers only look there. A robots.txt placed in a subfolder is ignored, so upload the generated file to your web root.
Does Disallow keep a page out of search results?
Not reliably. Disallow stops compliant crawlers from fetching a page, but a blocked URL can still be indexed if other sites link to it. To keep a page out of results, allow it to be crawled and use a noindex meta tag or header instead, or protect it behind authentication.
Is Crawl-delay respected?
Some crawlers honor Crawl-delay to slow their request rate, but Googlebot ignores it; you set Google crawl rate in Search Console instead. Bing and a few others do read it. The generator only adds the line when you set a value above zero.
Should I list my sitemap here?
Yes, it is good practice. A Sitemap line gives crawlers the absolute URL of your sitemap even if you have not submitted it anywhere, helping them discover your pages. You can list more than one sitemap, one per line.

More tools

JSON Formatter & ValidatorBeautify, minify, validate JSON with a collapsible tree view.Code Formatter & MinifierPretty-print or minify JS, TS, CSS, HTML, JSON, SQL and more.YAML ⇄ JSON ⇄ TOML ConverterConvert config between YAML, JSON and TOML, both ways.JSON Schema Viewer & ValidatorRender a JSON Schema as a tree and validate data against it.Markdown Editor (Live Preview)Write Markdown with a live, side-by-side HTML preview.Mermaid Live EditorTurn Mermaid text into flowcharts and diagrams, live.

More from the Hivly network

Free sister tools on our other sites.

Color Palette GeneratorShuffle palettes with the spacebar, lock favourites, copy hex codes.Length ConverterMetres, feet, miles, inches, km and more, instantly.Word CounterCount words, characters, sentences and reading time live.