AI search for shop owners · Lesson 2 of 6 · 3 min
How an AI assistant finds and reads your shop
AI companies send programs called crawlers to read web pages. Your robots.txt file and your server decide whether they get in.
Crawlers knock first
Before an assistant can mention your shop, one of its crawlers has to fetch your pages. Each crawler has a name (OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot, PerplexityBot and others) that it announces when it visits.
A well-behaved crawler first reads your robots.txt, a small text file at the root of your site that says which crawlers may read which pages. If robots.txt says no, it stays out.
Then your server has a say
Even when robots.txt says yes, a firewall, a security plugin or a CDN like Cloudflare can turn the crawler away. Since July 2025 Cloudflare blocks AI crawlers on new sites by default, a setting many owners never chose.
Do this
- Open yourshop.com/robots.txt in a browser and look for lines that mention AI crawlers, or let the free check read it for you, crawler by crawler.
Go deeper
Sources
- OpenAI: overview of its crawlers
- Anthropic: how its crawlers work
- Perplexity: its crawlers
- RFC 9309: the robots.txt standard
- MIT Technology Review: Cloudflare's default AI block