What Is AI Answer Variability?

AI answer variability is the way an AI tool gives different wording, names and links when you ask the same question more than once. For a health practice, it means a single screenshot of an AI answer shows one possible result, not a reliable picture of how you appear. The signal is in what repeats.

How AI answer variability works

An AI tool does not look up a stored answer. It writes the answer fresh each time, from the sources it decides to use in that moment. Small differences in what it retrieves or how it phrases things add up. Ask the same question twice and the wording will differ. Often the practices named and the links shown differ as well.

The best public measurement I have found is from Ahrefs. In a study published in November 2025, Ahrefs looked at over 43,000 keywords, each with at least 16 recorded AI Overviews, over about one month. They found AI Overviews have a 70% chance of changing between consecutive observations. They also found that 45.5% of cited sources were new from one observation to the next, because only 54.5% of the web addresses overlapped. Ahrefs notes that its checks were not daily, so the true change rate is likely higher. This is one study of one Google product, so I read it as strong evidence that Google AI Overviews move a lot, and as a reason to expect movement elsewhere.

The same study found something calming. The average similarity between consecutive responses was 0.95 on a scale where 1.0 is a perfect match. So the wording and the sources shift while the overall meaning stays close. For your practice, the main points a tool makes about your type of care are steadier than the wording and the exact list of names.

Why does the answer move? Google's documentation says AI Mode and AI Overviews may use different models and techniques, so the set of responses and links they show will vary. It also says both features may use a technique called query fan-out, which runs several related searches across subtopics and data sources behind one question. More searches mean more material, and the tool can pick different material each run.

Two ideas help here.

• The stable core. These are the names, sources and points that repeat across runs. They tell you what the tool tends to rely on.
• The volatile layer. These are the names, links and phrases that show up once and vanish. They are real, but a single appearance is not a trend.

Being eligible is also different from being shown. A page can be in the pool a tool draws from for a question and not appear in a particular run. Whether a source is linked is covered under AI citation.

The stable core is a working idea of mine and not a published finding, so I treat it as a method for reading results and not as a fact about how any vendor builds answers. It earns its keep because it points your effort at what repeats.

Example: AI answer variability at a health practice

This example is illustrative, not a transcript. A functional medicine clinic in a mid-size city asks an AI tool for the best functional medicine practices near them. On Monday morning the clinic is named second and the owner takes a screenshot. The owner sends it to the team as proof that the work has paid off.

On Tuesday the owner runs the same question five times in fresh chats. The results look like this.

• Run one names the clinic first.
• Run two names it third.
• Run three leaves it out and names four others.
• Run four names it second.
• Run five names it fourth.

The clinic appears in four of five runs. Two other practices appear in all five, and a third shows up once. The honest reading is that the clinic is in the pool and often named, but it is not in the stable core yet. The Monday screenshot was one draw from a range.

That is useful. The next step is to look at what the two practices that appear in every run have in common, such as a plain answer near the top of their service pages and consistent listings, and to compare. The clinic reruns the test weekly and tracks how many of five runs name it. Moving from four of five to five of five over a month means more than any single screenshot. The count across many questions is AI visibility.

How to check yours in 60 seconds

1. Pick one real patient question, such as "best [specialty] near [city]."
2. Open a new chat for each run, using the same tool and the same account on the same day.
3. Ask the question five times.
4. For each run, write down which practices are named and which sources are linked.
5. Mark which names appear in every run, which appear sometimes, and which appear once.

Repeat on another day. Keep patient details out of any AI tool while testing.

What to do about it

1. Stop reporting from single screenshots. Report the share of runs that name you.
2. Run each important question at least five times per tool, and note the date.
3. Separate the stable core from the volatile layer. Build around what repeats.
4. Study the practices that appear every time. Look at how their pages answer the question, and which facts they state.
5. Strengthen your own pages with a direct answer up front, as covered under prompt tracking for ongoing measurement.
6. Retest weekly and read the trend over a month or more.
7. Treat one miss as a prompt to run it again, not as a verdict.
8. Keep a simple sheet with the date, tool, question, number of runs and the names that appeared, so a month of results can be read at a glance.
9. Ask your team to bring you counts and not screenshots, for example "named in four of five runs," and agree on that wording before the first report.
10. Change one thing on your pages at a time and rerun the same questions afterward, so you can tell what moved the result.