Stax

…

Tools

robots.txt Generator

Generate a robots.txt that blocks chosen bots, including AI crawlers like GPTBot and ClaudeBot, adds your sitemap and downloads ready to upload. Free.

Last updated:

Bing and some other crawlers honour this. Google ignores it.

Disallow paths (for all bots)

A path needs to start with / (or *). One without it gets a / added.

Block specific bots entirely
Search engines

Blocking one of these removes your whole site from that search engine.

Link previews

Blocking these stops the preview card when your link is shared on that network.

AI training crawlers

These collect pages to train models. Blocking them does not remove you from AI search answers (see the next group).

AI search and assistants

Blocking these keeps your pages out of AI answers and citations. ChatGPT-User and Claude-User act for a person who asked, and some may not follow robots.txt.

SEO tools

Commercial crawlers that map your site for their customers.

robots.txt
User-agent: *
Disallow: /admin
Disallow: /api
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

Sitemap: https://example.com/sitemap.xml

Place this file at the root of your domain: https://yourdomain.com/robots.txt. It asks well-behaved crawlers to stay away; it does not secure anything.

Was this tool useful?

Use the tool above and you can rate it here.

How the Robots.txt Generator works

The robots.txt generator writes a robots.txt file for your site. Choose the paths to disallow for every crawler, block individual bots, set an optional crawl delay and add your sitemap addresses, and the tool shows the finished file to copy or download for your domain root. It adds the leading / to paths, drops a crawl delay or sitemap line that is not valid and tells you why, and warns you before you block a search engine.

User-agent groups and rules

Each group starts with a User-agent: line naming the crawler it applies to, and User-agent: * applies to every crawler that has no group of its own. A crawler follows the most specific group that matches its name. Within a group, Google applies the most specific rule, the one with the longest matching path, and Allow wins a tie. Paths are case-sensitive and must start with / or *.

AI crawlers: training versus search

The AI list is split by purpose. Training crawlers such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and meta-externalagent collect content for models, and one button blocks them all. Search and assistant bots such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User fetch pages to answer questions, so blocking them keeps you out of those answers. ChatGPT-User and Claude-User act for a person who asked, and OpenAI notes that robots.txt rules may not apply to ChatGPT-User.

Protecting private pages

Disallow: /admin keeps well-behaved crawlers away from a CMS back end, and a pattern such as /*.json$ excludes a file type. robots.txt is advisory only. It does not stop a person or a rude bot from opening a URL, and it shows the paths you list, so put real protection such as a login on anything sensitive. A disallowed page can still appear in search results as a bare address if other sites link to it.

Crawl-delay and Sitemap lines

Crawl-delay: 10 asks a crawler to wait 10 seconds between requests. Bing and some other crawlers honour it, while Google does not support the field. The Sitemap: line must be a full address and can appear anywhere in the file, more than once; it tells every crawler where your sitemap is.

Frequently asked questions

What is robots.txt?
robots.txt is a text file at the root of your website that tells search engine crawlers which pages or paths they are allowed or disallowed from crawling. It is a voluntary standard — well-behaved bots respect it, but malicious bots may ignore it.
Can I block AI training bots with robots.txt?
Yes, for crawlers that follow robots.txt. The main training crawlers are GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's Gemini), Applebot-Extended (Apple), CCBot (Common Crawl), Bytespider (ByteDance) and meta-externalagent (Meta). The older anthropic-ai token is still listed in case a tool uses it. Blocking training crawlers does not stop AI search products, which use separate bots.
Will blocking AI bots hide my site from ChatGPT or Claude answers?
Training and search are separate crawlers. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search; Anthropic uses ClaudeBot for training and Claude-SearchBot and Claude-User for search and user requests. Blocking the search and user bots keeps your pages out of those answers, and ChatGPT-User, which acts for a person, may not follow robots.txt. OpenAI and Anthropic document these names on their own sites.
Does Disallow: / block all bots?
Disallow: / for a specific User-agent blocks that bot from crawling your entire site. To block all bots, use User-agent: * followed by Disallow: /. Blocking Googlebot removes your site from Google Search, so the generator warns you when you tick a search engine.
What is Crawl-delay?
Crawl-delay asks a bot to wait that many seconds between requests. Bing and some other crawlers honour it; Google does not support it (Google's robots.txt documentation lists only user-agent, allow, disallow and sitemap). The generator accepts whole numbers from 1 to 3600 and leaves the line out otherwise.
Should I add my sitemap to robots.txt?
Yes. A Sitemap: line tells every crawler that reads robots.txt where your sitemap lives, in addition to submitting it in Google Search Console. It must be a full web address, and you can list several, one per line. The generator leaves out any line that is not a full address.

Related tools