No server access? How to see AI crawler visits anyway

The usual advice for checking AI crawler visits starts with "open your server logs". That's fine if you run your own server. It's useless if your site lives on Shopify, Webflow, WP Engine or Kinsta, where there is no log file for you to open. People in that position often conclude that crawler data is out of reach and fall back on guessing.

It isn't out of reach. It just comes from a different place. This post covers where, in the order we'd try them, and what each one can tell you.

Why it matters at all

If an AI engine never fetches a page, it can't cite that page from your site. If it fetches the page and gets an error, the same result follows with a different fix. You can't tell those cases apart from rankings or citation reports alone. The logs are the only place the difference shows up: a request, a user agent, a status code.

Without them, teams make changes blind. They rewrite content when the real problem was a 403 from a firewall rule.

Option 1: ask your host or CDN what they already keep

Before adding anything, check what exists. Many hosts keep some request data, even if it's not labelled "logs".

A CDN in front of your site (Cloudflare, Fastly, CloudFront and similar) usually holds request logs or analytics you can export.
Some hosts offer log downloads on higher plans, or on request through support.
Firewall or bot-management dashboards often list blocked requests by user agent. Even a blocked-bot list tells you which AI bots tried to arrive and were turned away.

The limit is detail. Dashboards give you counts, and counts hide the questions you care about: which page, which bot, which status code, and whether the claimed user agent was real.

Option 2: push logs out of the CDN

If your domain already runs through Cloudflare, you can send request logs to a destination you control. That setup, Logpush, ships logs from the edge, so it doesn't matter that the origin is a hosted platform. It's the standard way to get logs from a site where you can't touch the server.

You then need somewhere to put and analyse those logs, and a way to filter to AI user agents. Our list of AI crawler bot names gives you the strings to filter on: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and more.

Do the one thing people skip: verify by IP. A user agent string is a claim, and anyone can type "GPTBot" into a script. If your numbers include spoofed visits, you may think a page was fetched when it wasn't. The provider-published IP ranges are the reference, and they change, so the check has to use historical ranges as well as current ones, otherwise you drop real visits from ranges a provider has since retired.

Option 3: use a plugin on WordPress

If your site is on WordPress, including a managed host, a plugin can capture the requests from inside the application. It sees requests that reach WordPress, which means it won't see everything a CDN would, such as cached responses served at the edge. For many sites that's good enough to start, and the setup is a few clicks, not an infrastructure project.

Option 4: a tool that wraps the above

Setting up Logpush, storing the data, verifying the IPs and building filters is a small project. If you'd rather not own it, a platform that ingests the logs does the same work.

That's where we'd point you to Promptwatch, and it's our one recommendation for this problem. Its Agent Analytics feature shows real-time, IP-verified AI crawler logs, with spoofed user agents filtered out, error tracking with failure explanations, stackable path filters and a CSV export.

The setup details matter for hosted sites, and the Promptwatch changelog is specific about them:

The official WordPress plugin was released on September 10, installable from the WordPress Plugin Directory or as a ZIP during CMS setup.
Cloudflare Logpush became the default crawler log setup for WP Engine, Webflow, Shopify and Kinsta sites on October 6.
Verified crawler data now uses historical IP ranges from each provider, so crawler visits from ranges a provider has since retired are counted again, not dropped.

Promptwatch also supports other sources: AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN and custom HTTP, per the integration catalog. For a Shopify or Webflow store, the Logpush route is the one the changelog names.

The plan question

Crawler logs aren't on every plan, and we'd rather say so than have you find out after signing up. On the brand plans, crawler logs start on Professional at $245 a month, with 25M logs included. Essential, at $95 a month, has none. Business, at $579, includes 100M. All agency plans include crawler logs, starting with Kick-off at $199 a month. The free Explore plan doesn't include them either.

What the data lets you do

Once visits are flowing in, a few diagnoses become simple.

Check the first-fetch gap. Find a page you published two weeks ago. Has any AI bot fetched it? If not, look at linking and the sitemap before looking at content.

Check the errors. A cluster of 403 or 429 responses to one bot on one path usually means a firewall or rate limit. That's a configuration fix, not a content fix. Our post on checking whether AI bots can crawl your website covers the typical culprits.

Check what separates fetched from cited. A page fetched by ChatGPT-User and never cited has a different problem than a page never fetched. For that second diagnosis, see what a good crawler analytics tool should show beyond hit counts.

Keep the claim small

Seeing a bot visit a page doesn't mean the page will be cited. It means the engine had the chance to read it. That's still worth knowing, because without it you can't tell access problems from everything else.

If you're on a hosted platform and think crawler logs are impossible, the honest answer is that they need one extra step, and the step is a CDN log push or a plugin. Pick the route that matches your platform and check your plan before you start.