Can You Trust Your AI Citation Tracker? What a New Study Found

If you run a business in 2026, odds are someone has tried to sell you “AI visibility monitoring” — a dashboard that claims to show you whether ChatGPT recommends your business, which of your pages it cites, and how that’s trending week to week. The pitch is seductive because the fear is real: more buyers now start with an AI answer instead of a list of blue links. So of course you want to know if you’re in the answer.

Here’s the uncomfortable question nobody selling those dashboards wants you to ask: where is that data actually coming from? A new research preprint set out to measure exactly that, and the short version is that the stand-ins most tools rely on — Google’s search results and OpenAI’s developer API — miss most of what ChatGPT actually cites in its app. The paper, posted October 11 by researchers at the National University of Singapore and Xi’an Jiaotong University, was written up this month by Search Engine Journal.¹

This isn’t a reason to give up on measuring AI visibility. It’s a reason to measure it honestly — and to ask your vendor better questions. If you’re just getting your footing, it’s worth first reading how a Fort Wayne small business should actually start with AI, then coming back here for the measurement side.

Key takeaways:

• A new preprint recorded real signed-in ChatGPT web sessions and compared what ChatGPT cited against two common “stand-ins” — Google organic results and the OpenAI API. Combined, the two recovered only about a quarter of the exact pages ChatGPT cited.¹
• Google results alone matched just 6–9% of cited pages; the API alone matched roughly 23%. Matching improved to about half when you counted sites rather than exact pages.¹
• Citations are unstable run to run: about 40–45% of the pages ChatGPT returned for a question showed up in only 1 of 20 identical runs.¹
• Around the GPT-5.5 change, separate SISTRIX data showed German-language citations swinging 47% day to day, versus a normal 1–2%.³
• It’s a preprint, not peer-reviewed, and it did not test any specific commercial tracking tool — but it measured the exact data sources many of them lean on.¹
• The honest fixes: track at the site/brand level, run the same prompt many times, ask vendors whether their data is from the ChatGPT app or API, and re-baseline after every model update.

What did the new ChatGPT citation study actually measure?

The researchers wanted to answer a practical question: can you learn what ChatGPT cites without watching ChatGPT itself? Many visibility tools can’t see inside the ChatGPT app, so they approximate it — usually with Google’s organic results or with OpenAI’s developer API, both of which are easier to query at scale.

To test whether those approximations hold up, the team recorded the data flow inside their own signed-in ChatGPT web sessions so they could see the actual searches ChatGPT ran, the results it got back, and the pages it ultimately cited. They used 80 questions — 40 English and 40 Chinese — across four task types (recommendations, fact lookups, synthesis, and analysis), and ran each question 20 times on two models, GPT-5.3 Instant and GPT-5.5 Instant, per the arXiv preprint.² For comparison, they pulled Google organic results through SerpAPI and queried the OpenAI Responses API 20 times per question with web search turned on, all within three days of the matching ChatGPT run.

Two honest caveats before anyone quotes this as gospel. First, it’s a preprint — it has not been peer-reviewed, so treat it as strong early evidence, not settled science. Second, the authors did not test any named commercial tracking product; they tested the raw data sources those products tend to rely on. So the fair conclusion isn’t “your specific vendor is lying to you” — it’s “the plumbing a lot of these tools use is leakier than the marketing suggests.”

How much do the citation “stand-ins” actually miss?

A lot. And the gap is the whole point.

Measured at the exact-page level — did the stand-in surface the same URL ChatGPT cited? — the proxies came up short across the board. Here’s what the paper reported:¹

Read that top row again. If a tool infers your ChatGPT visibility primarily from Google rankings, it’s working from a view that matches well under one in ten of the pages ChatGPT actually cited. Even the OpenAI API — the closest official stand-in — caught only about a quarter of them, and combining both sources still left roughly three-quarters of ChatGPT’s exact citations unaccounted for.

The news is better if you stop obsessing over exact URLs. At the site level — counting whether the stand-in surfaced the same domain, even if a different page on it — Google plus the API recovered about half of the cited sites (48.7%–50.3%), and Google alone landed in the low-to-mid 20s.¹ That’s the single most useful takeaway for a business owner: the proxies are roughly twice as reliable at telling you whether your brand got cited as they are at telling you which exact page did.

This is also a clean illustration of why answer engine optimization is a different discipline from classic SEO. The paper’s own authors concluded that visibility work should separate “retrieval visibility” (did the engine fetch your page) from “citation visibility” (did it actually cite you), because the two don’t move together the way a Google ranking and a Google click do.

Why do ChatGPT citations swing so wildly week to week?

Because the thing you’re measuring barely sits still — even when nothing about your website changed.

The researchers ran each question 20 times under identical conditions and found that 44.5% of the distinct pages ChatGPT returned for a GPT-5.5 Instant question appeared in only one of those 20 runs (40.8% for GPT-5.3 Instant).¹ In plain English: ask ChatGPT the same question twenty times and nearly half the pages it touches show up exactly once. A single check is close to a coin flip.

Part of why is that ChatGPT doesn’t search with your words. In the runs with visible searches, GPT-5.3 and GPT-5.5 Instant never used the exact wording of the original question — they rewrote it into their own search terms first.¹ OpenAI’s own Responses API documentation describes the web search tool letting the model “access up-to-date information from the internet,” deciding on its own when and how to search.⁴ Different rewrite, different results, different citations.

Language adds another wrinkle that matters less locally but shows how much the machine improvises: for the Chinese-language questions, ChatGPT ran its searches only in English about 67–70% of the time.¹ The engine isn’t a stable lookup table; it’s a system making fresh decisions on every run.

Now layer model updates on top of run-to-run noise. Around the GPT-5.5 change in May, a separate SISTRIX analysis of 3.8 million German-language responses found day-to-day variation in citations jumped to 47%, against a normal range of just 1–2%.³ The average number of sources cited per answer slipped from 30.9 to 28.4, and individual domains lurched — Reddit citations rose 59% while Tripadvisor fell 53%. SISTRIX is careful to call that correlation, not proof the model change caused it, and the data is German-market only. But the direction is clear: when the model behind ChatGPT changes, the citation landscape can reshuffle overnight.

The business lesson is blunt. If your “AI visibility” dashboard shows a scary 30% drop week over week, the most likely explanation isn’t that your website fell out of favor — it’s that you’re looking at normal model noise, or a model update, misread as a trend.

How do you measure AI visibility honestly?

None of this means tracking is pointless. It means you track with the grain of how the system actually behaves instead of pretending it’s a tidy ranking report. Four rules carry most of the weight:

• Track at the site and brand level, not the exact page. Site-level matching was roughly twice as reliable in the study, and “is my business getting cited at all?” is the question that actually affects revenue. Chasing which specific URL won is mostly chasing noise.
• Run the same prompt many times before you believe a result. One check is near-random when 40–45% of pages appear in a single run out of twenty. Sample a handful of your real customer questions repeatedly and look at how often you show up across runs — a frequency, not a yes/no.
• Ask any vendor exactly where the data comes from. Is it from the ChatGPT app or the API? Do they pin the model version? Those two sources diverged badly in the study, so a tool that quietly substitutes API data for app behavior is measuring something different from what your customers see.
• Re-baseline after every model update. A swing right after a new model version is expected, not an emergency. Reset your benchmark and watch the new normal rather than reacting to the transition.

Honest measurement is slower and less dramatic than a dashboard with a big trend arrow, but it’s the only kind that won’t send you chasing ghosts. This is exactly the mindset behind sane ROI reporting: measure the thing that moves the business, with error bars, instead of a vanity number with a false decimal point.

What this means for a Fort Wayne business being sold “AI monitoring”

Picture an Auburn HVAC company whose owner just paid for a monthly tool that checks whether ChatGPT recommends them for “furnace repair near Fort Wayne.” Week one, the dashboard says they’re cited. Week three, the citation is gone and the number is flashing red. The owner panics, assumes a competitor out-maneuvered them, and is about to buy a bigger package to “fix” it.

Based on this research, the far more likely story is that nothing went wrong. The tool probably ran the prompt once or twice, ChatGPT rewrote the query differently that day, and the business landed in the ~45% of results that only appear in a single run — or a model update reshuffled the deck, the way SISTRIX watched happen across an entire market in a weekend. The honest move isn’t a bigger package; it’s running that same prompt ten times and asking whether the brand shows up most of the time.

That’s the lane we try to operate in for Allen County and Northeast Indiana businesses: setting an honest AEO baseline, sampling your real customer questions enough times to separate signal from noise, and reporting it at the brand level with the caveats intact. A dental office in Fort Wayne doesn’t need a dashboard that makes it jumpy every Monday — it needs a true read on whether it’s becoming the answer over a quarter, and a clear sense of what actually earns that. (If you want the flip side, we’ve written about how businesses actually earn AI citations in the first place, and whether your Fort Wayne business even shows up in Google’s AI answers today.)

Get an honest read instead of a jumpy dashboard

If you’re paying for AI-visibility monitoring and can’t answer “where does this number come from?”, that’s worth a second look — and it’s the kind of question we’d rather answer straight than sell around. Our answer engine optimization work includes setting a real baseline for how ChatGPT and Google’s AI treat your Fort Wayne business, sampled enough to be trustworthy and reported at the brand level with the noise acknowledged. If your current tracking is sound, we’ll tell you so. And when AI traffic does start converting, we’ll help you prove it — the companion to all this is whether Fort Wayne businesses can finally track their Google AI traffic. Reach out for a straight assessment.