PostHog
If your site logs AI crawler visits to PostHog, Lua reads that log once a day and shows the visits in Agent Analytics. This suits sites whose traffic does not pass through Cloudflare and that do not use a Vercel Drain.
AI crawlers do not run page scripts, so the PostHog snippet in your pages never sees them. The visit has to be logged on the server, before the page is sent.
1. Log crawler visits from your server
For each request whose user agent is a crawler, send one PostHog event:
| Field | Value |
|---|---|
event | ai_crawler_hit |
distinct_id | Any stable value, for example crawler:GPTBot |
properties.crawler_ua | The request’s full User-Agent |
properties.path | The requested path, for example /pricing |
properties.host | The requested host (optional) |
Lua identifies the crawler from crawler_ua, so your list of crawlers can be broad. Send the user agent as crawler_ua, not $raw_user_agent: PostHog drops events whose $raw_user_agent looks like a bot.
On Next.js, a middleware does this without slowing pages down for people:
// middleware.ts
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
const CRAWLER = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|GoogleOther|Applebot|bingbot|Amazonbot|Amzn-User|Amzn-SearchBot|Bytespider|CCBot|meta-externalagent|DuckAssistBot|YouBot|MistralAI-User|cohere-ai/i;
export function middleware(request: NextRequest, event: NextFetchEvent) {
const ua = request.headers.get("user-agent") ?? "";
const match = CRAWLER.exec(ua);
if (!match) return NextResponse.next();
event.waitUntil(
fetch("https://eu.i.posthog.com/capture/", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
api_key: process.env.NEXT_PUBLIC_POSTHOG_KEY,
event: "ai_crawler_hit",
distinct_id: `crawler:${match[0]}`,
properties: { crawler_ua: ua.slice(0, 512), path: request.nextUrl.pathname, host: request.nextUrl.host },
}),
}).catch(() => {})
);
return NextResponse.next();
}
export const config = { matcher: ["/((?!_next/|api/|.*\\.[a-z0-9]+$).*)"] };Use https://us.i.posthog.com/capture/ if your project is in the US region.
2. Create a personal API key
- In PostHog, open Settings > User > Personal API keys and click Create personal API key.
- Limit it to your project and give it the Query: Read scope.
- Copy the key.
Your project ID is under Settings > Project > Project ID.
3. Connect it in Lua
- In Lua, open Integrations and click Set up on the PostHog card.
- Choose the region, then paste the project ID and the key.
- Click Connect PostHog.
Crawler visits appear in Agent Analytics after the next daily sync.
What this source cannot show
- Status codes. The event is sent before the page is served, so Lua cannot tell a page that loaded from one that failed.
- Fake crawlers. Anyone can claim to be GPTBot in a user agent, and a log cannot check the claim.