What decides whether AI cites you is the retrieval crawlers, not the training crawlers. Across nine days of logs, training crawlers (ClaudeBot, GPTBot) visited more often than retrieval crawlers (Googlebot, Bingbot, PerplexityBot, OAI-SearchBot) — but the former feed training data, are never retrieved, and produce zero citations.
Reporting on "total AI crawler volume" gives you a badly misleading answer. And submission is not indexing: there is a gap of days to weeks between them.
One: nine days of measured data
From 6 to 14 October 2026 we audited our own server access logs day by day. The log is JSON, one record per request, capturing timestamp, source IP, User-Agent, path, status code and response size.
| Date | Valid external requests | Real AI crawler visits |
|---|---|---|
| Oct 6 | 70 | 0 |
| Oct 8 | 165 | 13 |
| Oct 10 | 323 | 15 |
| Oct 12 | 291 | 75 |
| Oct 14 | 155 | 45 |
The differences within the same set of crawlers are large. On 12 October, the peak day, the 75 AI visits broke down as:
- 39 retrieval-type: Googlebot 22, Bingbot 14, PerplexityBot 2, OAI-SearchBot 1
- 36 training-type: ClaudeBot 31, GPTBot 5
In other words, on a day that looks strong on a naive AI-traffic dashboard, nearly half the traffic was never going to help.
Two: retrieval vs training, side by side
This is not speculation. Each vendor documents what its crawler is for.
| Crawler UA | Operator | Purpose | Produces citations? |
|---|---|---|---|
OAI-SearchBot | OpenAI search | Live retrieval for ChatGPT answers | Yes |
PerplexityBot | Perplexity | Live retrieval | Yes |
Googlebot | Search index | Yes (AI Overviews, citations) | |
Bingbot | Bing | Search index | Yes (Copilot, Bing answers) |
Bytespider | ByteDance | Search index | Yes (Doubao answers) |
ClaudeBot | Anthropic | Model training crawl | No |
GPTBot | OpenAI | Model training crawl | No |
Training crawlers collect content for model training, and training data is not retrieved at inference time and does not appear in chat answers. Being crawled 31 times by ClaudeBot does not mean Claude will mention you once.
Three: training crawlers mostly fetch robots.txt and the sitemap
We logged the exact path of every AI request. On 12 October:
| Crawler | Hits | Paths requested |
|---|---|---|
| ClaudeBot | 31 | robots.txt ×12, sitemap.xml ×12, other ×7 |
| GPTBot | 5 | sitemap.xml ×1, other pages ×4 |
| Googlebot | 22 | en/news/overseas-geo-b2b-supplier-selection.html, en/news/why-ai-wont-cite-you.html, en/news/ai-search-brand-visibility.html, china-geo-agency.html, 404.html |
| Bingbot | 14 | index.html, feed.xml, sitemap.xml, pricing.html, geo.html |
The pattern is clear: 24 of ClaudeBot's 31 requests (77%) were only for robots.txt and sitemap.xml. It learned the site existed and took it away for training. It did not read a word of the articles.
Googlebot, by contrast, fetched concrete article URLs. That is the evidence of content actually being read.
Implication: without a correct sitemap, training crawlers show up, learn you exist and leave, while retrieval crawlers may never find your content at all. With a good sitemap, retrieval crawlers read page by page.
Four: what we did, and what we can honestly claim
On 14 October 2026 we completed Google Search Console domain-property verification and sitemap submission. Pre-submission checks: HTTP 200, Content-Type: text/xml, all 37 URLs verified reachable. We closed the Bing loop the same day: verification file deployed, sitemap submitted via API (returned Status=Success, UrlCount=37), plus six key URLs batch-submitted.
Being precise about the limit: submission is not indexing. There is typically a gap of days to weeks between them. As of 14 October we can confirm only that the submission completed. We cannot claim indexing. Treating submission as indexing is the most common form of self-deception in this field.
We will keep monitoring real status through the Bing Webmaster API and the Search Console index query, rather than guessing.
Five: five things you can do today
1. Split retrieval from training in your logs. At minimum track OAI-SearchBot, PerplexityBot, Googlebot, Bingbot and Bytespider separately. GPTBot and ClaudeBot volume is essentially uncorrelated with your citation count.
2. Confirm robots.txt allows the retrieval bots. This is the most common miss and the cheapest thing to fix.
3. Treat the sitemap as your primary entry point. In our data, 77% of training-bot requests landed on it. No sitemap means near-invisibility to AI engines.
4. Keep monitoring after submitting. A successful submission does not mean indexed. Both Google and Bing expose index status APIs. Use them.
5. Do not optimise for training crawlers. Rewriting pages for GPTBot is wasted effort: it never enters retrieval results. Spend the effort on content quality and third-party sources off-site.
Six: how to reproduce this yourself
Step 1 — extract AI crawlers from access logs. Mainstream bot User-Agent strings are fixed and recognisable. Your log needs UA, path, status code and source IP.
Step 2 — cross-check by IP and behaviour. Single IPs with very high frequency deserve suspicion. Flag "same UA across many IPs" and "one IP at very high frequency" separately so you can spot traffic farms.
Step 3 — strip internal traffic. Monitoring scripts, curl checks and scheduled jobs all land in your access log. On 14 October, 291 of 446 requests (65%) were internal or self-generated. Without this step your AI visit count is inflated roughly threefold.
Step 4 — bucket by retrieval vs training. Every conclusion above comes from this step.
FAQ
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's training crawler. Content it collects enters model training data, is not retrieved at inference time, and never appears in chat answers. OAI-SearchBot is the search retrieval crawler, invoked live each time a user asks a question. Only what it retrieves can show up in a ChatGPT answer.
Is ClaudeBot crawling useful for GEO?
Essentially no. ClaudeBot is Anthropic's training crawler and takes no part in live retrieval. In our nine-day log it visited more often than several retrieval crawlers, yet produces no citations.
Does submitting a sitemap mean the site is indexed?
No. Submission and indexing are separate events with a gap typically measured in days to weeks. A successful submission only proves the action completed. Index status has to be confirmed through the Google Search Console and Bing Webmaster Tools indexing query interfaces.
How do you tell retrieval crawlers from training crawlers?
By User-Agent string. Retrieval crawlers include OAI-SearchBot, PerplexityBot, Googlebot, Bingbot and Bytespider. Training crawlers include GPTBot, ClaudeBot and CCBot. Counting them in one bucket produces a badly misleading result.
Why does internal traffic need to be excluded from log analysis?
Because monitoring scripts, curl checks and scheduled jobs all land in the access log. On 14 October, 291 of 446 requests (65 percent) were internal or self-generated. Without excluding them, reported AI visit volume is inflated roughly threefold.
Data note: this article uses our own server access logs from 2026-10-06 to 2026-10-14, output by a self-built audit system. The analysis script and criteria are reproducible. Service and product details are as published on our site.
AI assistance disclosure: data preparation and the first draft were assisted by AI and reviewed by a human before publication. The underlying data comes from raw logs and is verifiable.
Want your brand to show up in AI answers?
We have turned this method into a deliverable service: sitemap construction, AI crawler allow-listing, search engine submission, and monthly indexing monitoring.
See the GEO service