AI Visibility Tracker: Six Identical Prompts Returned 18 Different Sources

An AI visibility tracker sounds straightforward. Ask an assistant your category questions, record whether your brand appears, chart it over time.

The problem is upstream of the recording. The thing you are measuring does not hold still. Send the same prompt to the same model twice and you get two answers that overlap partly, not fully. Record one of them and you have recorded one sample from a distribution, then presented it as a fact.

We wanted to know how big the spread actually is, so we ran the experiment rather than assuming it. The short version: six identical requests produced eighteen cited sources, and no single source was cited by all six.

This article covers what we ran, what came back, and the four fields an AI visibility tracker has to store if the number is going to be worth looking at next month.

What we ran

Two experiments, both on prompts a SaaS team would actually use for category research.

Experiment 1: one prompt, one model, six runs. Model gpt-4o, web search enabled, United States, English. Prompt: "What are the best rank tracking tools for a small SaaS team in 2026? Give a short list with sources." Six separate live calls, run back to back within the same session window.

Experiment 2: two models, same prompt, three runs each. The identical prompt was sent to Claude Sonnet 5 and to Gemini 3.8 Flash, three times each, with no web search. Without search we are measuring which products the model names from its own knowledge, which is a separate signal from which sources it cites.

We also sent a second prompt through the web-search path four times, to see whether citation behaviour is consistent across question types: "How do I track AI Overviews for my website? Give concrete methods with sources."

Six runs is not a large sample, and we treat it that way. It is enough to show the direction and rough size of the variance, which is the point.

Result 1: eighteen sources, zero shared by all six runs

The six web-search runs cited 18 distinct URLs between them. Here is how often each domain appeared.

Not one domain was cited by all six runs. Not one URL was cited by all six runs.

The run-to-run overlap is the number that matters here. Comparing every pair of runs, the mean Jaccard overlap on cited URLs was 0.167. Two of the fifteen pairs had an overlap of exactly zero, meaning those two answers shared no source at all.

A source cited in five of six runs is the ceiling we observed. Nothing was cited every time.

There is a second finding buried in that table. The domains the model reached for are mostly not the well-known comparison sites in the category. Nine of the eighteen domains appeared exactly once, and several are small sites with content that reads as assembled for the query rather than reported. For a brand, this cuts both ways. It means the citation slot is easier to win than a traditional "best tools" ranking. It also means the slot is being filled by whoever produced a page that matched the query shape that run, and it will be refilled next run.

Result 2: the answer itself changes size

The citations moved, and so did the length of the answer.

Output length ranged from 505 to 1,108 tokens, a spread of 603 tokens, or about 73 percent of the mean. The longest answer was more than twice the shortest. Five of the six runs landed in a fairly tight band between 813 and 1,108 tokens; one run stopped early at 505 and cited five sources instead of nine to fourteen.

For measurement this matters in a specific way. If your tracker captures "does the brand appear in the answer, and at what position", a short run has fewer slots to appear in. A team that samples once a week and happens to draw a short run will record a worse result than a team that drew a long one, with no change on the website.

Result 3: some prompts get no citations at all

The second prompt, about tracking AI Overviews, went through the same web-search path four times and returned zero citations in every run.

The answers were stable in length, varying by only 8 percent across four runs. But the model answered from its own knowledge and named no sources, so there was nothing to record in a citation column.

This produces a specific and misleading tracker reading. A citation-rate dashboard would show zero for this question, which looks like a visibility failure. It is not. It is a question shape that does not trigger source lookup. The right reading is "citation not applicable", and a tracker that cannot distinguish that from "citation checked and you were absent" will send you chasing a problem that does not exist.

Result 4: swapping the model is a different measurement, not a repeat

With web search off, we measured which products each model names. That isolates the model's own recommendation set from whatever the search index returned that minute.

Claude Sonnet 5 named four to five products per run. Across three runs it named six distinct products: AccuRanker, Ahrefs, SEMrush, SE Ranking, Serpstat, and SerpWatcher. Three of those appeared in all three runs: AccuRanker, Ahrefs, and SEMrush. Mean pairwise overlap 0.587.

Gemini 3.8 Flash named two to five products per run. Across three runs it named seven distinct products: AccuRanker, Ahrefs, Looker Studio, Nightwatch, SE Ranking, SEMrush, and Wincher. One of those, SE Ranking, appeared in all three runs. Mean pairwise overlap 0.306.

Between the two models, four products were named by both and five were named by only one.

The same question produces a partly different recommendation set on each model and each run.

The practical consequence: if you track "AI visibility" as one number, you are averaging across at least two independent sources of variation, the run and the model. A brand that shows up reliably in one assistant and not in another has a real, actionable finding. A brand whose single-sample measurement moved by two positions has noise.

What an AI visibility tracker should record

Four fields turn a screenshot into a measurement.

The run count, not the run result. Store every run and report the range. "Cited in 4 of 6 runs" is a fact. "Cited, position 3" is a coincidence. Six runs is a reasonable floor for a small program; twelve is better if the prompt matters commercially.

The citation field separately from the mention field. "The model recommended us" and "the model linked to us" are different outcomes with different fixes. Our six runs produced 18 citations from 18 different URLs; a tracker that only records brand mentions would have captured none of that churn.

The answer channel state. Record whether the run used web search at all, and record the answer length. A zero-citation run and a zero-mention run look identical in a dashboard and mean opposite things.

The prompt text, verbatim, with a version. Prompt wording drives which sources get pulled. If the prompt changes between months, the trend line is broken. Version the prompt the way you version a tracking script.

Building the run loop

Manual checking does not survive contact with a calendar. The loop is a script that runs a prompt N times, stores each raw response with its citations, and diffs the current window against the last one.

This is a good fit for an agent because the work is repetitive, rule-bound, and needs a written record. In Claude Code or Codex, the useful instruction is to keep the raw runs on disk and never overwrite them, then report a summary table rather than a single figure. If you already pull Search Console and Bing data through MCP servers, the same session can hold the citation log next to the impression data, which is where the two numbers start being useful together. Our notes on what those servers expose are in what four SEO MCP servers actually do, and the presence side of the same problem is covered in our forty query AI Overview snapshot.

One design rule matters more than the rest: the tracker's output should be a distribution. Report cited-in-4-of-6, not cited. Report the source list, not the top source. Any dashboard that compresses six runs into one number has thrown away the only information the extra runs bought.

If the goal is competitor comparison rather than monitoring, a rank-tracking loop over time is the closer instrument, and the data-source tradeoffs are in building a rank tracker from two sources.

Limits of a six run sample

Three honest caveats.

The sample is small. Six runs bounds the spread loosely. It establishes that overlap between runs is partial and that zero-overlap pairs occur; it does not give you a confidence interval for your own category.

The results are time-bound. These runs were made on September 12, 2026 against live models. Both the models and the underlying search results change.

The cited domains are a sample of one commercial prompt. A different question shape, such as a comparison question or a how-to question, will produce a different citation pattern. The useful move is to run the same design on your own ten most commercially important prompts and record what your category does.

FAQ

How many runs do I need before an AI visibility number is meaningful? Six is the practical floor for a single prompt, and the number should be reported as a count out of runs rather than a position. If a prompt drives revenue decisions, twelve runs gives a tighter read for a few dollars.

Does this mean AI visibility tracking is not worth doing? It means single-sample tracking is not. The variance is the finding. A tracker that reports "cited in 4 of 6 runs, sources changed by 12 domains since last month" is describing a real system that a competitor is competing in.

Why did one prompt return zero citations? The model answered from its own knowledge rather than performing a source lookup. That is a property of the question, not a failure of your site. Track it as not applicable rather than as zero.

Should I track ChatGPT, Claude, and Gemini separately? Yes. In our three-run comparison the two models overlapped on only four of nine named products. Averaging them hides a program-level decision about which assistant is worth optimising for first.

Can I reduce the variance by lowering the model temperature? Partly, and it is worth testing on your own prompts. But the citation set is also driven by the search index, which you do not control. Run count is the more reliable lever.

Auspia view: if you are building AI visibility tracking this quarter, build the run log before the dashboard. The dashboard is a rendering choice. The run log is the measurement. Start with ten prompts, six runs each, stored verbatim, and you will learn more in one week than a year of single-sample checking will tell you.

Author: Ethan Marlowe, GEO Measurement Lead Across 500+ Prompts at Auspia. Ethan writes about prompt tracking, citation reports, and the visibility dashboards that survive contact with a real measurement cycle.