Crawlable and Still Not Cited: What Sits Between the Two

The ai search visibility gap is the distance between an AI crawler being able to read your page and an AI answer choosing to cite it. We measured the first half on our own site and passed. We measured the second half and failed. This post reports both, and it is honest about the part nobody, including us, has tested yet.

Most advice on this topic treats access and citation as one problem and then offers fixes for access. Our own numbers show they are separate. A site can be read perfectly by every AI crawler and still be absent from every answer.

Can AI crawlers read our site?

Yes, completely. On 21 September 2026 we fetched utilityseo.com and one blog post as OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Googlebot, with a normal browser as the control. All six requests returned HTTP 200. The responses were byte identical across all six agents: 153,350 bytes for the home page and 102,371 bytes for the post. Both pages are server rendered, with one h1 each, 1,463 and 1,495 words of text in the raw HTML, and 3 and 8 blocks of JSON-LD structured data respectively.

The method and the exact user agent strings are in We Fetched Our Own Site as Four AI Crawlers, so anyone can repeat it in ten minutes. This is one site on one date. It proves delivery and nothing else.

This diagram highlights the ai search visibility gap where content is technically crawlable yet fails to appear in AI generated citations.

Are we cited in AI answers?

No. On 6 September 2026 we put 40 buying questions about SEO tools to a search-grounded AI assistant and recorded which brands each answer named. UtilitySEO was named in 0 of the 40. Semrush was named in 92% of answers and Ahrefs in 82%. The questions, answers and counts are in Which SEO Tools Do AI Assistants Recommend.

So the gap on our own site is not theoretical. Every crawler gets the full page, and no answer uses it.

The keyword table flags which tracked keywords show an AI Overview, and whether the site is cited in it.

Why is being crawled not the same as being cited?

Crawling decides whether a model can read a page. Citation is a separate selection step: an answer engine retrieves many candidate pages and uses a few. Google's documentation for its own AI features describes pages being shown as supporting links in AI Overviews and AI Mode, and says there is no special optimisation required beyond normal search eligibility (Google: AI features and your website). That tells you what makes a page eligible. It does not tell you what gets it chosen.

What does get a page chosen is where the evidence runs out. The SEO Meta Study (Kresimir Corluka, 10 September 2026) reviewed 9,249 SEO documents and found 10 entries on AI visibility and 7 on answer formatting, against 36 on content depth and 20 on title tags. Controlled tests of generative engine optimisation are close to absent (The SEO Meta Study). Confident lists of citation factors are, for now, mostly opinion.

What have we ruled out?

Our site is a useful counter-example to three common pieces of advice, because it already does all three.

Structured data. Both pages we tested carry JSON-LD: 3 blocks on the home page, 8 on the post. We are not cited. Google's documentation for AI Overviews and AI Mode says no special schema.org structured data is needed to appear in them (Google: AI features and your website). Structured data may still help in ways nobody has measured, but it is plainly not sufficient on its own. llms.txt. We publish one at utilityseo.com/llms.txt (1,849 bytes, live). We are not cited. The same Google page says you do not need new machine-readable or AI text files for its AI features, and we know of no controlled test showing that llms.txt changes citation anywhere else. Crawler access. Every major AI crawler gets the same complete server-rendered page a browser gets. Access is solved for us, and it did not produce citations.

None of this proves these things are useless. It shows that on one site, having all three did not close the gap.

What have we not measured yet?

Who Perplexity cites instead of us. For our main unbranded query, we have not yet published a dated capture of Perplexity's answer and its sources. Until we do, this post makes no claim about which sites it cites or how many of them are competitors. That capture is the next test in this series. Whether changing a page changes its citation. Nothing here is a controlled test. We have one observation of access and one set of 40 answers. A real test changes one thing on a set of pages, leaves a matched set alone, and checks both against the same prompts over several weeks. How fast answer sets turn over. If the sources behind an answer change weekly, a single capture says little. We do not yet know the turnover rate.

How can you test your own gap?

1. Check access first. Fetch a key page with each AI crawler's user agent and compare status code, byte length and structured data count against a normal browser. If they differ, fix that before anything else.
2. Record citations on a fixed prompt set. Write 20 to 40 questions a buyer would ask without naming you. Run them in the assistants you care about, record every brand and URL cited, and date the capture.
3. Repeat on a schedule. Run the same prompts again after four weeks without changing anything, so you learn the natural turnover before you judge any change.
4. Change one thing at a time. If you change a page, change one element on some pages and leave comparable pages alone. Otherwise you cannot tell what caused a difference.
5. Publish the result either way. Most published SEO results are wins; the SEO Meta Study found 81.9% positive outcomes against 3.8% negative, which says more about what gets written up than about what works.

Summary

The ai search visibility gap is real, and our own site is in it: fully readable by every AI crawler we tested, carrying structured data and an llms.txt, and named in 0 of 40 answers from a search-grounded AI assistant. That rules out access as the explanation for us and shows that the usual technical fixes are not sufficient on their own. What actually decides citation is barely tested anywhere, so treat any confident checklist, including ones we have published, as a hypothesis. We will report the Perplexity capture and the first controlled test in this series when we have them, whatever they show.

Sources

• Google: AI features and your website
• Kresimir Corluka: The SEO Meta Study
• UtilitySEO: We Fetched Our Own Site as Four AI Crawlers
• UtilitySEO: Which SEO Tools Do AI Assistants Recommend

Free AEO Scan: checks how ready a site is to be cited by AI answer engines.