Our A/A Test Had a Winner
It shouldn't have.
Published
It shouldn't have.
We are testing a change to the ialoc booking widget. Nothing fancy. But before running the A/B test, we ran an A/A test.
Wait, what's an A/A test? 2 arms, the exact same experience, nothing to detect. It's a check on the measurement system itself. If one of the arms wins, something in the setup is broken.
Ours had a winner.
The first one ran for a week with PostHog (person-level, client-side flag, Bayesian experiment view). The test arm came out ahead by about 1 percentage point on the booking funnel. And it led on all 6 days. Exposure counts were balanced, so the traffic split was fine. Identical experience, skewed result. Hmm…
So we started digging.
We checked what we could from the data we already had: exposure counts, the flag's edit history, interference from a parallel experiment, and random re-splits of the same users. We found no sample-ratio mismatch, no change to the flag and no interference from the other test.
And the gap survived every re-split we tried. So it wasn't plain luck. It pointed at the assignment mechanism.
Our main suspect was identity. The first setup assigned the arm client-side, using a user ID that rotates daily and is different between cookieless embed loads and consented first-party loads. But results were counted per person. If the ID at exposure and the identity behind a later conversion didn't line up the same way in both arms, you get a steady bias. Like the one we saw.
Did we prove it? No.
Proving it meant a proper investigation. More queries, more instrumentation, more days. And we had other stuff to ship.
This is the part I'll defend: we did not need the root cause. We had an instrument that failed a test it should have passed. That's enough to stop using it. You don't keep a thermometer that shows 39 degrees when you feel fine… even if you can't explain why it's wrong.
Ok, so what did we change?
3 things:
A. Assignment moved server-side. A random draw per load, independent of any analytics identity.
B. The unit of measurement became the widget load instead of the person. Conversions are matched back to the same load.
C. The analysis was fixed before the test started. One read at the end. No peeking. (I know. It's hard.)
The second A/A ran the same week we relaunched. The difference between the arms this time? 0.05 percentage points. And the leading arm changed from day to day. Boring. Which is exactly what you want from an A/A test.
So what's the lesson?
Run an A/A test before you trust an A/B test. Sounds obvious. And yet…
Statistical methods reason about the data they receive. Bayesian, fixed-horizon frequentist, sequential, doesn't matter. None of them can tell you, on their own, that the data came from a broken assignment mechanism. Or that the unit of analysis doesn't match the unit of exposure. Those are properties of the setup. And the only way to find them is to test the setup itself.
The second lesson is less comfortable. Knowing has a cost. The investigation we skipped would have taken days we didn't have. The A/A test let us make the right call without it. You don't need to explain the mechanism to see that the instrument is lying to you.
Part of me still wants to know why. The engineer in me hates leaving it open. But that's my problem, not the business's.
We launched the experiment on the new setup. The results arrived and we deployed the new version for everybody to enjoy. And now? Now we started another A/B test on the new version. It never ends :)
Have you ever run an A/A test? Did yours have a winner too?
If you found this worth reading, share it on Bluesky or LinkedIn, or follow me on /dev/null.