Good morning. Something worked last quarter. You still can't say what.
Growth.
The Controlled Experiment
When your numbers moved last quarter — the conversion rate that lifted, the churn that eased — what precisely were you comparing them against, and could you rule out the season, a competitor's outage, the campaign that launched the same week and the fact that everyone knew you were watching? Every one of those is still sitting inside the figure you reported, and there is no arithmetic that takes them back out.
Ronald Fisher spent the 1920s at Rothamsted, an agricultural research station north of London, working out why decades of field trials had produced so little anyone could trust. His answer, set out in The Design of Experiments in 1935, was randomisation. Assign the treatments by chance and every difference you never thought of — soil, drainage, weather, the farmhand's mood — spreads itself evenly across both groups instead of hiding inside your result. He illustrates the machinery in the second chapter with a lady who declares she can taste whether the milk went into the cup before the tea or after. Fisher doesn't argue with her. He designs eight cups, four made each way, presented in an order she can't anticipate, and settles in advance how many she would have to get right before he'd believe her. Rothamsted lore holds that she was a colleague and that the test was really run; the book claims neither, and the design is the part that matters. Ron Kohavi, Diane Tang and Ya Xu carried it online in Trustworthy Online Controlled Experiments in 2020, and nothing essential changed: a group that gets the thing, a group that doesn't, and chance deciding which is which. The trap is the before-and-after chart. Compare this month with last month and you've credited yourself with the season, the competitor's outage and everything else the world did while you weren't looking.
Members only · 4 principles + template + AI mentor
364 more frameworks are waiting.
The Controlled Experiment is Day 1 of 365. One framework every morning for a year — across strategy, sales, negotiation, leadership, and more.
$1 /day
Billed $365/yr · cancel any time
365 frameworks
12 topics · templates · AI mentor
One payment a year · the whole library · no auto-renewal.