Search indexer or user fetch: reading AI crawler logs
Not every AI bot visit means the same thing. Here is how to tell indexing from a live fetch, and how the Nov 4 GEO 360 change shifts the crawl totals you report.
Open an AI crawler report and the first instinct is to add up the visits. More crawls, more interest, good news. That instinct is wrong often enough that we stopped using the raw total as a headline.
The reason is that AI bots do different jobs. Some build an index so the engine can find your pages later. Some fetch a page at the moment a person asks a question. A visit from the first kind says you are in the engine's catalog. A visit from the second kind says a page was read to answer someone. Those are different events, and a report that blends them misleads.
Two kinds of visit
Search indexers. These crawl the web ahead of time so an AI search product has something to retrieve from. OAI-SearchBot, Claude-SearchBot, and Meta-WebIndexer belong here. A visit tells you the page is being collected. It does not tell you that anyone asked about it.
User fetches. These happen while an answer is being produced. ChatGPT-User is the example: it fetches a page while answering a user. That is a citation crawl, a page being pulled in service of a live question.
There are other bots, including those that gather training data, and we keep them out of this comparison. The point here is the split between indexing and answering.
What changed on November 4
Promptwatch changed how GEO 360 classifies crawler traffic. Per the changelog, search indexers such as OAI-SearchBot, Claude-SearchBot, and Meta-WebIndexer now count as AI crawls. Citation crawls are only bots that fetch a page while answering a user, like ChatGPT-User.
The consequence is easy to miss. Citation crawl totals fall by as much as AI crawl totals rise, and page categories can shift. If you have a monthly chart, expect a step on the change date that reflects reclassification, not a change in what bots did. Annotate it before a client or a boss asks why the line moved.
How to read the two numbers
We look at the pair, not either one alone.
- AI crawls high, citation crawls low. Indexers are collecting the page, but few live answers fetch it. The page may be indexed and not yet useful for any question you track.
- Both present on the same URL. The page is collected and also being pulled into answers. This is the pattern you want to protect.
- Citation crawls on a page indexers skip. Unusual. Check whether access rules or a firewall treat the two bot families differently.
- Neither. Check access first. A robots.txt rule or a CDN rule may be turning bots away.
None of these is a verdict on visibility. They are prompts to look somewhere.
Tying crawls to citations
A crawl alone tells you little. What matters is whether a crawled page is then cited. Promptwatch's Agent Analytics logs AI crawler visits, verified against each operator's published IP ranges, so spoofed user agents are filtered out. That filter matters, because a log full of fake bot names makes every conclusion unreliable.
The part we use most is the crawl-to-citation path, with a citation rate per page. It separates two failures that look alike in a dashboard. If a page is crawled often and rarely cited, you probably have a content problem: the bot reached it and the engine did not find it useful for the question. If a page is rarely crawled, you probably have an access or discovery problem.
Error tracking in the same view gives failure explanations, which saves time when a bot is getting blocked. Filters stack and persist, and a CSV export covers the cases where a developer wants the raw rows.
A sensible review routine
- Note the November 4 reclassification on every chart that crosses it.
- Break crawls into indexer visits and user fetches. Report them separately.
- List the pages you care most about and check each for both kinds of visit.
- For pages with crawls and no citations, read the page as the engine would, and compare it to what the answers cite instead.
- For pages with no crawls at all, check robots.txt and the CDN before touching content.
We keep this short on purpose. A checklist that takes an hour will not get done.
What we do not claim
We do not say that more indexer visits will lead to more mentions. We do not claim a ratio between crawls and citations is healthy at any given level. We have no benchmark for that and we distrust anyone quoting one. What we offer is a way to find which problem a page has, so the fix is aimed at the right place.
If you want to see your own crawler logs split this way, Promptwatch has a 7-day free trial. If you want us to read them with you, write to hello@visiblyrank.com.