Free SEO Tool

Robots.txt Generator

Create a clean robots.txt file for search engines and add your sitemap URL for better crawling.

FreeNo login requiredFreeSeoCheckers

Create robots.txt

Sitemap URL(s) One per line, optional Crawler Rule Extra Disallow Paths One per line, optional Crawl-delay Seconds, optional - most bots ignore this

AI Crawlers Block AI training/scraping bots individually - regular search engines are unaffected

GPTBot (OpenAI) ChatGPT-User Google-Extended (Gemini training) CCBot (Common Crawl) ClaudeBot (Anthropic) anthropic-ai PerplexityBot Bytespider (ByteDance) Amazonbot Applebot-Extended

Note: LLMs.txt is the emerging standard for giving AI crawlers permission and guidance rather than just blocking them - see our LLMs.txt Generator.

Output

How to use this Robots.txt Generator

Robots.txt is the first file most search engine crawlers request when they visit your site. Getting it wrong is one of the fastest ways to accidentally block Google from indexing pages you actually want ranked — this tool builds a safe, correctly formatted file in a few seconds.

1. Add your sitemap URL

Paste the full URL to your XML sitemap (e.g. https://example.com/sitemap.xml). This helps crawlers discover your pages faster and is one of the most common recommendations in a technical SEO audit.

2. Choose a crawler rule

Pick "Allow all crawlers" for most public websites, "Block all crawlers" only for staging/development sites you don't want indexed, or "Allow all but block admin/private folders" to keep login and admin areas out of the index without hiding your real content.

3. Copy and upload

Copy the generated text and upload it as robots.txt to your site's root directory, so it's reachable at example.com/robots.txt. Search engines will not read it from any other location or subfolder.

Popular SEO Tools

SEO Checker · Technical SEO Checker · Robots.txt Generator · XML Sitemap Generator · Schema Generator · Keyword Density Checker · Redirect Checker · Favicon Generator

Frequently Asked Questions

What is robots.txt?

Robots.txt tells search engines which areas of your website they can crawl. It lives at the root of your domain and is checked before most other files on a crawl.

Should I block all crawlers?

Usually no. Block crawlers only for private or development websites — blocking a live site by mistake is one of the most common causes of a sudden drop in search traffic.

Does robots.txt stop a page from appearing in Google?

Not reliably. Robots.txt only blocks crawling, not indexing — a blocked URL can still appear in search results without a description if other sites link to it. Use a noindex meta tag instead if you want a page fully out of search results.

Do I need to list every page in robots.txt?

No. Robots.txt is for rules (allow/disallow), not a page list — that's what your XML sitemap is for. Reference your sitemap URL inside robots.txt so crawlers can find it automatically.

Where exactly should robots.txt be uploaded?

At your domain root only — example.com/robots.txt. A robots.txt file placed in a subfolder (like example.com/blog/robots.txt) is ignored by search engines.