AI crawlers and robots.txt: should you allow GPTBot, ClaudeBot and Google-Extended?
Blocking a training crawler is not the same as blocking a search crawler. Learn what each AI bot does, and which ones take you out of AI answers when you block them.
Last checked:
AI companies don’t crawl the web with a single bot. One crawler collects content to train models, another builds the search results shown inside the chat, and a third only shows up when a user asks the assistant to open a link. Block them all with a blanket “no AI” rule and you may keep your content out of training data, but you also give up your chance of being cited in ChatGPT and Perplexity answers.
This guide sorts the major platforms’ crawlers using their own documentation and suggests a sensible starting configuration for ecommerce and B2B sites.
What does each AI crawler do?
The table below lists the user-agent tokens each platform documents. The value in the “User-agent” column is the name you use in robots.txt. These lists change, so check the linked sources for the current version before you edit your file.
| User-agent | Company | What it does | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for model training. | Your content stays out of future model training. Your visibility in ChatGPT search is not affected. |
| OAI-SearchBot | OpenAI | Builds the search results ChatGPT shows and links to. | ChatGPT is far less likely to cite you as a source. |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT to open it. | ChatGPT can’t read the page the user asked for. This is a user request, not a crawl. |
| ClaudeBot | Anthropic | Collects content for model training. | Your content stays out of Claude model training. |
| Claude-SearchBot | Anthropic | Builds the search results Claude uses in answers. | Claude is far less likely to cite you as a source. |
| Claude-User | Anthropic | Visits a page when a user asks Claude to open it. | Claude can’t read the page the user asked for. |
| Google-Extended | A control token, not a separate crawler. It governs whether your content is used to train Gemini models and to ground their answers. | Google Search, AI Overviews and AI Mode are not affected. Those depend on Googlebot. | |
| Googlebot | Feeds Google Search, including AI Overviews and AI Mode. | You disappear from Google Search and from Google’s AI results. | |
| PerplexityBot | Perplexity | Crawls sites for Perplexity’s search results. | Perplexity is far less likely to cite you as a source. |
| Applebot-Extended | Apple | Controls whether your content is used to train Apple’s AI models. | Siri and Spotlight search are not affected. They rely on Applebot. |
The most common mistake: blocking search along with training.
Plenty of sites copy an “AI bots” blocklist and disallow every entry. When that list includes search crawlers, the site quietly removes itself from the answers where ChatGPT or Perplexity recommend products and link to sources. Treat it as two separate decisions: should my content train models, and do I want to be cited in AI answers?
The second mistake happens on the Google side. Blocking Google-Extended does not remove you from AI Overviews: Google builds those from its regular search index via Googlebot. If you don’t want to appear in AI Overviews, Google’s controls are snippet rules such as nosnippet, and those also affect your regular search snippet.
The third mistake lives outside robots.txt. CDNs and security services such as Cloudflare offer one-click settings that block AI crawlers. If that setting is on, bots can’t reach you even when robots.txt allows them. Check your server logs to see which bots actually get through.
A starting configuration for ecommerce and B2B sites.
If your site sells something, being recommended in AI answers is usually an opportunity. A sensible default is to keep search crawlers open and decide on training crawlers separately. In the example below, training crawlers are blocked while search and user-triggered bots are allowed. To allow training as well, delete the first two rules.
- User-agent: GPTBot → Disallow: /
- User-agent: ClaudeBot → Disallow: /
- User-agent: Google-Extended → Disallow: / (Google Search and AI Overviews are unaffected)
- User-agent: OAI-SearchBot, Claude-SearchBot, PerplexityBot → Allow: /
- User-agent: * → disallow paths that don’t need crawling, such as cart, account and internal search pages.
- End the file with a Sitemap: line pointing to your XML sitemap.
How to check that the change worked
After you update robots.txt, confirm the file loads at your site root, for example yoursite.com/robots.txt, with a 200 status code. Then filter your server or CDN logs by user agent: requests from allowed bots should continue, and requests from blocked ones should drop off.
Marvedo’s free instant analysis checks, for any single page, whether your robots.txt lets AI crawlers in. Whether you actually get cited in AI answers is a separate question: track it with the same set of questions for a few weeks after you open access.
This guide was prepared with AI assistance using official sources. Examples are not customer results and do not promise performance improvements.