AI search crawlers vs AI training crawlers: which to block
Each AI company now runs separate agents: one that finds and cites pages for answers, one that opens a page when a person asks about it, and one that collects training data. You can block training and still be found.
Let these in to be found
| OAI-SearchBot | OpenAI | AI search and answers | docs |
| ChatGPT-User | OpenAI | Opens pages when a person asks | docs |
| Claude-SearchBot | Anthropic | AI search and answers | docs |
| Claude-User | Anthropic | Opens pages when a person asks | docs |
| PerplexityBot | Perplexity | AI search and answers | docs |
| Perplexity-User | Perplexity | Opens pages when a person asks | docs |
| Googlebot | AI search and answers | docs | |
| Applebot | Apple | AI search and answers | docs |
| bingbot | Microsoft | AI search and answers | docs |
| DuckAssistBot | DuckDuckGo | Opens pages when a person asks | docs |
| MistralAI-User | Mistral AI | Opens pages when a person asks | docs |
Your choice: training crawlers and tokens
| GPTBot | OpenAI | Model training | docs |
| ClaudeBot | Anthropic | Model training | docs |
| Google-Extended | Model training | docs | |
| Applebot-Extended | Apple | Model training | docs |
| meta-externalagent | Meta | Model training | docs |
| CCBot | Common Crawl | Model training | docs |
Google-Extended and Applebot-Extended are robots.txt tokens, not separate crawlers: they control whether content fetched by Googlebot or Applebot may be used for AI training, without affecting search.
Check your shop free. It takes about 20 seconds and tells you exactly what’s blocked and the lines that fix it.
Check my shop