Why AI Answers Change Every Time (and How Mention Rate Fixes It)

AI answers change because the model picks words with some randomness and, when it searches the web, the pages it reads can change between runs. SparkToro and Gumshoe found that the same prompt gave the same brand list in fewer than 1 in 100 runs. So one check tells you almost nothing. The fix is to ask each prompt several times and measure your mention rate: the share of runs that name or cite you.

This guide explains why answers move, why a single screenshot is noise, how to calculate mention rate, and how to read the change from week to week.

TL;DR

• AI answers vary between runs, so one check tells you very little.
• In a SparkToro and Gumshoe study, the same prompt gave the same brand list in fewer than 1 in 100 runs.
• Ask each prompt several times and track the share of runs that name you.
• Compare week over week, not one run with the next.

Why does the same prompt give different answers?

There are two main reasons.

The model picks words with some randomness

A language model writes one word (or token) at a time. At each step it has a list of likely next words, each with a probability. Most chat products do not always pick the top word. They sample from the list. This makes answers sound natural, but it also means two runs can take different paths. One path lists five tools. The next lists six, in a different order, and drops one.

The web search step changes too

When ChatGPT or Gemini searches the web before answering, the answer depends on which pages come back. Search results can shift by the hour. The engine may also write a different search query each time. Different pages in means different brands out.

What the data says

SparkToro and Gumshoe ran the same prompts many times and compared the brand lists. The same prompt gave the same list of brands in fewer than 1 in 100 runs. That is not a small wobble. It means the answer you saw this morning is very likely not the answer your buyer will see this afternoon.

Why is one check just noise?

Picture a prompt where your brand appears in about half of all answers. You check it once. You have a coin flip's chance of seeing yourself. If you see your brand, you may think you are winning. If you do not, you may think you are invisible. Both are wrong.

Now picture your team checking by hand, once a week. The result jumps around. Someone writes a new page, the next check shows your brand, and the page gets the credit. It may have done nothing at all.

Single checks cause three common mistakes:

1. False wins. You see your brand once and stop working on the prompt.
2. False alarms. You miss once and rewrite a page that was fine.
3. Wrong credit. You link a change in the answer to work that did not cause it.

How does mention rate work?

Mention rate turns many noisy answers into one number you can track.

Mention rate = runs that name or cite your site ÷ total runs × 100

Here is an example with 5 runs of one prompt in one engine:

You were named or cited in 3 of 5 runs, so the mention rate is 60%. You were cited (linked as a source) in 2 runs, so you have 2 cited runs.

Mention versus citation

These are two different signals:

• Mention: the answer names your brand in its text.
• Citation: the answer links to your site as a source.

A mention with no citation often means the engine knows you from other sources, like lists and reviews. A citation means one of your pages was used to build the answer. Both matter. Track them apart.

What counts as visible?

You need a line between "visible" and "not visible". A simple and useful line is 50%. If you show up in at least half of the answers, most buyers who ask that prompt will see you. Below 50%, most will not.

How many runs do you need?

More runs give a steadier number, but each run costs AI usage. Here is how the step size changes with the run count:

Three runs is a rough signal. Five is a fair default for most prompts. Ten is better for the prompts that matter most to your revenue.

How do you read week-over-week change?

This is where most teams go wrong. A mention rate is still a sample. It moves even when nothing real has changed.

Know the size of one run

With 5 runs, one run is 20 points. A drop from 60% to 40% is one answer out of five. That can easily be noise. Do not rewrite a page because of it.

Look for changes that last

A real change usually holds. Use these rules of thumb:

• One week, one prompt, one run of change: note it, do nothing.
• Two or three weeks in the same direction: look into it.
• A group of related prompts all moving together: this is a real signal. Something changed, either in the engine or in the sources it reads.

Check the cited domains

When your rate drops, look at which other domains the AI cites for that prompt. If a new review site or Reddit thread appears in many runs, that source may be shaping the answer now. Your fix may be to get on that source, not to edit your own page.

Compare engines apart

Do not average ChatGPT and Gemini into one number. They search different sources and change at different times. Ahrefs found only about 11% of domains are cited by both ChatGPT and Perplexity, which shows how much engines differ.

Read competitor share

Your mention rate means more next to your rivals. If you are at 40% and a competitor is at 90% on the same prompt, you have a clear gap. If every brand is at 30%, the engine is spreading answers wide, and the prompt may be less settled.

How does AgenticSEO measure mention rate?

AgenticSEO (our product) uses this method in its weekly AI visibility loop:

1. Prompt list. It lists the prompts buyers type into ChatGPT and Gemini, and estimates demand from Google volume (DataForSEO) or Search Console impressions.
2. Repeated runs. It asks each prompt several times. The default is 5 runs per prompt, and you can set 1 to 10. ChatGPT runs go through the OpenAI API with web search. Gemini runs use Google Search grounding.
3. Results. For each prompt it shows your mention rate, cited runs, competitor share, and the other domains the AI cites.
4. Visible or not. A prompt counts as visible at 50% or more.
5. Gap tickets. Prompts where you are not visible become tickets, mapped to the best page to improve or a new page to write.

Runs per prompt depend on your plan: 3 on Free and 5 on the per-site plan. The AI checks are included in the plan. See plans and run limits.

A note on scope: today AgenticSEO tracks ChatGPT and Gemini. Perplexity, Claude, Copilot and Google AI Overviews are on the roadmap, not built yet. Answers through an API may also differ from what a logged-in user sees in the chat app, so treat any tracker as a sample of the answer space, not a copy of every user's screen.

A simple weekly routine

If you track AI visibility by hand or with a tool, this routine keeps you honest:

• Run each tracked prompt at least 5 times per engine.
• Record mention rate and cited runs, not a yes or no.
• Mark each prompt as visible (50% or more) or not.
• Compare to last week, but act only on changes that last 2 to 3 weeks or hit a group of prompts.
• Check the domains the AI cites for prompts you lost.
• Turn real gaps into one task each: a page to fix, a page to write, or a third-party source to work on.

Track your first 10 prompts on the free plan. Each one is asked several times, so you see a share of answers, not a single reply. Start free

Related reading

• 8 AI search ranking factors that matter (and 4 that do not)
• How to get cited in ChatGPT
• Mention rate in the AgenticSEO docs