The AI crawlers we check
18 crawlers and robots.txt tokens from 11 companies, last updated 2026-10-21. Each one links to its operator’s own documentation. New crawlers are researched every week and added after review.
| Token | Company | What it’s for | Follows robots.txt | Confirmed |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | AI search and answers ChatGPT search | yes | not yet |
| ChatGPT-User | OpenAI | Opens pages when a person asks ChatGPT (pages opened on a user's request) | unclear | not yet |
| GPTBot | OpenAI | Model training Model training | yes | not yet |
| Claude-SearchBot | Anthropic | AI search and answers Claude search | yes | not yet |
| Claude-User | Anthropic | Opens pages when a person asks Claude (pages opened on a user's request) | yes | not yet |
| ClaudeBot | Anthropic | Model training Model training | yes | not yet |
| PerplexityBot | Perplexity | AI search and answers Perplexity search index | yes | not yet |
| Perplexity-User | Perplexity | Opens pages when a person asks Perplexity (pages opened on a user's request) | unclear | not yet |
| Googlebot | AI search and answers Google Search, including AI Overviews | yes | not yet | |
| Google-Extended robots.txt token only | Model training Gemini training and grounding (robots.txt token, not a separate crawler) | yes | not yet | |
| Applebot | Apple | AI search and answers Siri, Spotlight and Safari suggestions | yes | not yet |
| Applebot-Extended robots.txt token only | Apple | Model training Apple foundation model training (robots.txt token, not a separate crawler) | yes | not yet |
| bingbot | Microsoft | AI search and answers Bing and Copilot answers | yes | not yet |
| DuckAssistBot | DuckDuckGo | Opens pages when a person asks DuckAssist answers | yes | not yet |
| MistralAI-User | Mistral AI | Opens pages when a person asks Le Chat (pages opened on a user's request) | unclear | not yet |
| meta-externalagent | Meta | Model training Meta AI training and indexing | yes | not yet |
| Amazonbot | Amazon | Several uses Alexa and Amazon services | yes | not yet |
| CCBot | Common Crawl | Model training Open web archive widely used for AI training | yes | not yet |