AI crawlers · Apple
Is your site blocking Applebot-Extended?
Applebot-Extended is Apple's robots.txt token for model training. Check your site now: we read your robots.txt exactly as Applebot-Extended must, ask your server as each AI crawler, and look at your product pages.
Does blocking it hurt?
Model training Blocking Applebot-Extended only opts your content out of Apple's model training. It doesn't remove you from Apple's AI answers (that's Applebot). Whether to allow training is your choice, not a fault.
In our research, 2% of 2,159 online shops block Applebot-Extended in robots.txt. See the research.
Allow or block it in robots.txt
Applebot-Extended is a robots.txt token, not a separate crawler: Apple reads pages with its normal crawler and checks this token to decide whether they may be used for training.
To let it in (add to your robots.txt, at yourshop.com/robots.txt):
User-agent: Applebot-Extended Allow: /
To keep it out:
User-agent: Applebot-Extended Disallow: /
A more specific group wins: if your robots.txt also has a group for Applebot-Extended, that group decides, not the “User-agent: *” rules. Our check reads it the same way (RFC 9309).
Other places it gets blocked
- Cloudflare. Cloudflare has blocked AI crawlers by default on new sites since July 2025, and it can also write your robots.txt for you. If your site uses Cloudflare, check its AI crawler settings first (MIT Technology Review).
- Security plugins and firewalls. WordPress security plugins and host firewalls sometimes refuse crawlers they don’t recognise, even when robots.txt allows them. Our check asks your server as each crawler and shows any refusal.
Where this comes from
Apple’s own documentation: support.apple.com. Apple: disallowing Applebot-Extended opts content out of training Apple's models; Applebot still crawls for search.
Apple’s other crawlers
- Applebot (AI search and answers)
We check 18 AI crawlers in one go. See them all · How the check works