In one sentence: generative answers vary between identical runs, so a trend is only meaningful if the question set is fixed, the sample is large enough to average over the variance, and the gaps where nothing was collected are visible rather than filled in.
Where the variance comes from
Ask the same engine the same question twice and you can get different brands, in a different order, citing different sources. Nothing about your site changed between the two. This is the property that breaks every measurement habit carried over from search, where a SERP is broadly stable within a day.
A single run is an anecdote. The unit of measurement is a rate over a question set, not an observation.
Sampling that absorbs the noise
- Average across questions, not across runs of one question. Twenty-five questions run four times tells you more than one hundred questions run once, and costs the same.
- Keep the set and the engines fixed. Any change to either makes the before and after incomparable, however tempting the new question is.
- Anchor results to the run's start timestamp, not to when the job finished. A slow run otherwise lands in the wrong period and puts a kink in the trend that nobody can explain later. We shipped this bug: runs taking over ten minutes were never snapshotted at all.
What counts as a real movement
With a set of twenty-five to forty questions, single-digit percentage changes between runs are usually within the noise. The signals worth acting on are:
| Signal | Why it is more trustworthy than a percentage |
|---|---|
| The same direction across two consecutive runs | Variance is not usually directional twice |
| A change concentrated in one intent group | Noise spreads evenly; a real cause does not |
| A named competitor appearing or disappearing consistently | An event with a cause you can go find |
| A change in the citation source mix | Reflects where the engine looks, which moves slowly |
Alerting without crying wolf
Require two consecutive runs in the same direction before raising anything. One run is noise and everybody learns to ignore alerts that fire on noise. Also check the trigger logic itself: an alert that fires on a level rather than on an edge will re-fire every run while the condition persists, which trains people to mute it. We shipped that bug too.
The rule that protects the whole series
A run that failed is not a zero. A week with no collection is not a week of zero visibility. Rendering either as 0% turns a flat trend into a fake decline, and quietly dropping them raises the average instead. Both are worse than an honest gap in the line, because a gap prompts the right question and a wrong number does not.