overlift.

32,363 real randomised experiments from the Upworthy Research Archive, re-split so the arm you would have picked gets scored on traffic it never saw. Every number below is measured on real clicks — no effects are simulated.

Pick a winner, then check it against held-out traffic

Half of each arm's impressions choose the winner; the other half scores it. Re-split to see how much of the verdict was the coin, not the copy.

—

Headline	Impressions	CTR	Selection half	Held-out half	Shrunk

—

lift you would have claimed

—

lift on held-out traffic

—

shrunk prediction

—

p-value, winner vs rest

—

What the dashboard would have shown you along the way

The z-statistic for the top two arms as traffic accumulates over ten looks. Inside the grey band nothing is significant at 5%; the dashed line is the O'Brien-Fleming boundary that keeps ten looks honest.

—

Peeking, on real A/A splits

One arm dealt into two. True difference: exactly zero.

False positives when nothing is there.

Arms, on real null tests

One arm dealt into k. Every arm identical.

A “significant” winner where none exists.