How we check

Every result comes from your own public pages, read the way AI crawlers read them, and is decided by fixed, published rules. No AI judges your shop.

1. robots.txt

We fetch your robots.txt and apply it exactly as the Robots Exclusion Protocol (RFC 9309) requires: the most specific group that names a crawler wins, and within it the longest matching rule. We do this for each of the 18 AI crawlers and tokens in our public register, last updated 2026-10-21. We also spot when Cloudflare manages your robots.txt, because then the fix belongs in Cloudflare.

2. Your server’s response

We ask for your page once as a normal browser and once as each AI crawler, using its published user-agent. If the browser gets the page and a crawler is refused or challenged, a firewall, CDN or security plugin is blocking it.

3. The raw page

We read the HTML your server sends, without running JavaScript, because most AI crawlers don’t run it. We measure how much readable text is there, and check for noindex and “noai” signals.

4. Product data

We find a product page (from your home page links or sitemap) and read its schema.org Product markup: name, price, currency, availability, image, brand, return policy and reviews.

How the answer is decided

• Visible: every AI search and answer agent can reach and read the page.
• Partly visible: some are blocked, or the page is readable but missing key product data.
• Not visible: the agents that power AI search are blocked, the page asks not to be indexed, or it is empty without JavaScript.

Blocking training-only crawlers is your choice and never counts against you.

Honest limits

• robots.txt results are exact: it is a public file and the rules are standard.
• Server results are an indication. We ask from our own address, not the crawler’s, and some firewalls treat the real crawlers differently.
• Being visible is the first condition for being recommended, not a promise that you will be.
• We only read public pages and never log in, fill in forms or buy anything.

Our research checks

To publish the state of AI visibility, we also check popular public shops, chosen from the Tranco research ranking of websites, across South Africa, the United States, Europe and the rest of the world. A shop’s market comes from its country domain, or else the currency on its product pages. These checks identify themselves as Khanyo-Research, obey each site’s robots.txt for every page, read only robots.txt, the home page and one product page, and never pretend to be an AI crawler. Results are published only as totals across many shops; no shop is named, and we don’t contact shops about them. To opt out, disallow Khanyo-Research in your robots.txt (we also obey rules written for our old name, SeenByAI-Research).

Keeping the crawler list current

AI companies add and rename crawlers every few months. Every day we check each operator’s own documentation page for changes, and every week new crawlers are researched and added only when the operator documents them on its own website. Third-party lists don’t count as evidence.