Robots.txt Generator

Build a valid robots.txt to tell search engines which paths to crawl or avoid, set a crawl delay and point to your sitemap. Copy it to your site's root. Generated in your browser.

100% free
Private & safe
Runs in your browser

This tool runs entirely in your browser and needs JavaScript enabled to work. The full guide, examples and FAQ below are available without it.

Presets

User-agent

Crawl-delay (optional)

Allow/Disallow rules (one per line)

Sitemap URL

robots.txt

User-agent: *
Disallow:

Test a URL path against your rules

Important: robots.txt ≠ privacy

Disallowing a page only asks crawlers not to fetch it — it does not prevent indexing (a blocked URL can still appear in results if it's linked elsewhere) and does not protect private content. To keep a page out of Google, use a noindex meta tag or header; to protect data, require authentication.

Why use Robots.txt Generator?

Robots.txt Generator helps you optimise and publish with confidence, producing clean, standards-friendly output you can paste straight into your site.

Runs instantly in your browser — nothing to install, nothing uploaded.
Completely free, with no account, no watermarks and no usage limits.
Output: robots.txt
Set the user-agent.
Works on any device — desktop, tablet and phone — and is fully keyboard accessible.

How to use Robots.txt Generator

Set the user-agent.
Add allow/disallow rules.
Add your sitemap URL.
Copy the robots.txt to your site root.

Examples

Blocking /admin from crawlers.
Pointing Google to your sitemap.

Supported formats & limits

Output: robots.txt

Limitations

Disallow is a request, not enforcement — it doesn't secure private pages.

Frequently asked questions

Does Disallow hide a page?

No — it asks crawlers not to index it; use auth to truly protect pages.

Where does robots.txt go?

At your domain root, e.g. example.com/robots.txt.

Uploaded?

No — local only.

Was this tool helpful?

XML Sitemap GeneratorBuild a valid XML sitemap: dedupes, validates, checks the domain, splits at 50,000 URLs and downloads.htaccess Redirect GeneratorGenerate .htaccess redirect rules for URLs, HTTPS and www.Open Graph Meta Tag GeneratorGenerate Open Graph and Twitter Card meta tags.Schema.org JSON-LD GeneratorGenerate Schema.org structured data (JSON-LD).