You submitted your sitemap. Why don't AI crawlers read your articles?

1. The overestimated sitemap: it is only a list

Almost everyone who publishes content assumes one thing: upload your URLs to a sitemap and crawlers will come. That is not wrong — a sitemap genuinely solves the question of which pages exist. What it does not solve is whether those pages are worth reading, or how a crawler would get to them in the first place.

Our access logs give a clear enough contrast to settle this. Between 2026-10-20 and 10-23, AI crawlers whose source IPs fall inside each vendor's officially published network ranges made 345 requests. Where those 345 requests landed is a more useful signal than any opinion.

The takeaway: crawlers spent nearly half their effort reading the table of contents, and almost none reading the content.

This does not mean crawlers were not interested — a genuinely indifferent crawler would not have re-read the sitemap 167 times. The more accurate reading is that crawlers treat a sitemap as a validation list rather than a crawl plan. They need internal links to understand site structure, weigh page importance, and decide how deep to go. A sitemap only tells them which URLs exist.

It is worth being precise about where a sitemap stops and internal links begin.

• What a sitemap does do: tell crawlers which URLs exist on the site, when each was last updated, and their relative priority. It is the crawler's main route to a map of the site, and for a new site with no inbound links it is often the first hint that a page exists at all.
• What a sitemap does not do: express any relationship between pages. An article and a product page look identical inside a sitemap, so a crawler cannot infer from the list that the article's subject and the product page's subject are the same topic.
• What a sitemap never does: carry a content-quality signal. A sitemap with 50 URLs still leaves the crawler to decide which to read first — and it decides based on link structure, not list order.

2. The broken link graph: why articles are not reached

The next check is more direct. Before the fix, our 12 articles (6 Chinese and 6 English) had only 26 internal link edges between them. And all 13 Chinese business pages plus all 12 English business pages — 25 in total — had zero inbound links to any article.

Zero inbound links is a far more serious problem than having too few articles. Put yourself in the crawler's position: you see six article links on /news.html, so you fetch five of them — and across those five fetches you read one article body, because the other five have equally thin inbound links. And if an article's inbound link count is zero, it never even reaches your candidate queue.

Three specific points of disconnection

1. Orphan articles: our article on when AI crawlers actually arrive had zero inbound links in both languages before the fix. It appeared in the /news.html list, but no other page pointed to it — and for a crawler, a link from a listing page carries far less weight than an inline link inside body content.
2. No path between business pages and content pages: /geo.html, /overseas.html, /faq.html and /pricing.html carry the site's most authoritative topic signals, yet none of them contained a single article link. The topic signal and the content assets were structurally disconnected.
3. Missing or unreadable anchor text: the site had almost no inline links. Even where a "related reading" link existed, it needed descriptive anchor text — "this article explains the difference between GEO and SEO" carries meaning, while a bare URL carries none.

All three point to the same conclusion: content is not written, it is wired together. The body of an article determines whether it is worth reading; the link structure determines whether it ever gets read. The first was already fine (22 pages pass the readability audit: average body length 8,765 characters, canonical tags present, no noindex). The second was empty.

3. llms.txt: a factual list written for the model

llms.txt is a convention that has been widely adopted since 2024: a plain-text file at the site root telling large language models what this site is, what falls inside the citable scope, and which pages matter. Its division of labour with a sitemap is clear — a sitemap gives crawlers URLs, llms.txt gives a model facts.

Our llms.txt had a typical flaw: it described entity information, service boundaries and legal positions in detail, but never listed the articles. To a model, that is the same as saying the service scope is clear but the articles themselves are invisible. The result was that across 345 real crawler requests, llms.txt was read only 3 times — crawlers had no idea what to do with it.

Three structural changes you can execute

1. Put the article list into llms.txt: one line per article, with title, absolute URL and a one-sentence description. The description should be a complete statement of what the article covers and what distinct conclusion it reaches, not a tag. Six lines of text cost almost nothing, and they move a model's candidate citation set from zero to six.
2. Rebuild the article ring: string the articles into a ring following topic logic (definition → comparison → diagnosis → measurement → selection → measured data), with each article's previous/next pointers pointing at its neighbours and the ring closed. After the fix, orphan articles went to zero.
3. Give business pages inline article links: add a further-reading block to 25 business pages, four to six topically relevant articles each, with anchor text written as "title — one-sentence description". This connects the commercial pages' topic authority to the content assets.

What these three changes share is that none of them adds a single word of content. They only change how content is connected. That is the central claim of this article: while total content volume is still modest, structural work has a higher marginal return than adding more content.

4. What changed after the fix

All three changes were completed on our site on 2026-10-23. Below is the before-and-after comparison of structural metrics, all reproducible from the public pages and our own audit script.

These are structural metrics, not citation metrics. A change in citation rate only becomes visible once search engines and AI engines complete a re-crawl, which typically takes one to four weeks. Structural work solves reachability; whether a reachable page is then cited depends on the distinctiveness of the content and the authority of its sources — a separate question.

The limits of this measurement should be stated plainly. The observation window was four days, and 345 requests is not a large absolute volume. The server was also stable throughout, so the sample does not cover a complete crawl-peak cycle. The conclusions apply to the question of reachability, and should not be extrapolated into a quantified citation-rate forecast.

• Every figure in this section comes from the same log snapshot and the same judging script, and is reproducible by a third party using public means; the script and criteria are described in the article.
• The before-and-after comparison uses two independent measurements taken before and after the change, with no content edits in between, so the difference is attributable to the structural work itself.
• Better structure and better citation rate are not mechanically linked. The latter is also driven by content distinctiveness, source authority and engine algorithms, and this article makes no quantified claim about it.

5. A reachability self-check for your own site

The table below is ready to use. Every item can be verified with public means, without any paid tool.

If any single item in this list comes back "no", your content is in a state of "written but not delivered". These six checks and content quality are two independent lines. Our content-quality audit (body length, canonical tags, structured data, FAQ blocks) had already passed in full — but if reachability fails these six, AI engines still cannot read the pages.

The order matters too: fix reachability before adding more articles, because a new article published into a broken link graph inherits the same invisibility. Reachability first, then volume — otherwise every new article becomes another orphan.