AI crawlers · OpenAI
Is your site blocking OAI-SearchBot?
OAI-SearchBot is the crawler behind ChatGPT search. Check your site now: we read your robots.txt exactly as OAI-SearchBot must, ask your server as each AI crawler, and look at your product pages.
Does blocking it hurt?
AI search and answers Blocking OAI-SearchBot keeps your shop out of ChatGPT search: your pages can't be read there, so they can't be shown or recommended.
In our research, 1% of 2,159 online shops block OAI-SearchBot in robots.txt. See the research.
Allow or block it in robots.txt
OpenAI says OAI-SearchBot follows robots.txt, so the lines below decide whether it reads your shop.
To let it in (add to your robots.txt, at yourshop.com/robots.txt):
User-agent: OAI-SearchBot Allow: /
To keep it out:
User-agent: OAI-SearchBot Disallow: /
A more specific group wins: if your robots.txt also has a group for OAI-SearchBot, that group decides, not the “User-agent: *” rules. Our check reads it the same way (RFC 9309).
Other places it gets blocked
- Cloudflare. Cloudflare has blocked AI crawlers by default on new sites since July 2025, and it can also write your robots.txt for you. If your site uses Cloudflare, check its AI crawler settings first (MIT Technology Review).
- Security plugins and firewalls. WordPress security plugins and host firewalls sometimes refuse crawlers they don’t recognise, even when robots.txt allows them. Our check asks your server as each crawler and shows any refusal.
Where this comes from
OpenAI’s own documentation: developers.openai.com. OpenAI's crawler overview lists OAI-SearchBot for ChatGPT search results, controlled separately from GPTBot.
OpenAI’s other crawlers
- ChatGPT-User (Opens pages when a person asks)
- GPTBot (Model training)
We check 18 AI crawlers in one go. See them all · How the check works