Sample frameDay 343·Growth · Compound

Your last test came back inconclusive. That result was settled before the test ever ran.

Growth.

The Minimum Detectable Effect

5 min read·Apply by lunch

The question

For the test you're most impatient to launch this quarter, do you know how many people would have to pass through each variant before a lift the size you're honestly hoping for could register as anything other than noise? If you don't, the test will still finish, it will still produce a chart, and you will still read a decision into it.

The idea

Jacob Cohen spent most of his career as a professor of psychology at New York University, and in 1962 he did something nobody had bothered to do. He took the 1960 volume of the Journal of Abnormal and Social Psychology and asked of each study, one by one, whether it had ever been capable of finding the effect it went looking for. On average it hadn't: a typical study had less than an even chance of detecting a medium-sized effect even when the effect was really there. His answer was Statistical Power Analysis for the Behavioral Sciences, published in 1969 and much expanded in a second edition in 1988, which sets out the arithmetic binding four quantities together — your sample size, the size of the effect, how confident you demand to be that a result isn't chance, and the probability you'd catch a real effect if one existed. Fix any three and the fourth is decided for you. Run that arithmetic before the test rather than after it, and the useful output is the minimum detectable effect: the smallest true lift your traffic can tell apart from noise on the settings you've chosen. A flat result from a test that never had the power is not an answer to the question you asked. It's an answer to a different one — how many people did you send.

The framework

01
Work out the detectable lift before you write the variant.
Four numbers get agreed before anything is built: your current conversion rate, the traffic you can realistically send through each variant in the time you'll allow, the significance level you'll accept and the power you want. Feed them into any power calculator and it hands back the minimum detectable effect. Do this while you're still deciding what to build, not once the results have landed, because the answer regularly kills a fortnight of engineering before a line of it is written. Most teams run the calculation only afterwards, to explain a disappointing result they've already had.
02
How big a win are you hoping for?
The calculator answers a question you have to ask honestly first: what size of lift would justify the build. Teams say any lift is a lift, then design tests for effects of one or two per cent, which on almost any realistic traffic are invisible. If the minimum detectable effect comes back at fifteen per cent and you'd be delighted with three, you haven't got a test, you've got a coin toss with a dashboard attached. Write both numbers next to each other and the decision makes itself.
03
The quadrupling rule — halve the effect, quadruple the traffic.
Sample size scales with the inverse square of the effect you're chasing, and that single fact governs your whole roadmap. Detecting a lift half as large takes roughly four times the traffic. On the standard settings of five per cent significance and eighty per cent power, a page converting at three per cent needs somewhere near fifty thousand visitors per variant to see a ten per cent relative lift, and around four times that to see a five per cent one. Which is why small sites should test big structural changes and leave button colours to businesses with millions of visitors.
04
An underpowered test does damage in both directions.
It misses real wins, which is the obvious cost. It also throws up the occasional large, exciting and entirely spurious result, because when a test can only see huge effects, whatever clears the bar has to look huge. So put the minimum detectable effect at the top of every test document and read every flat result against it. If the test couldn't have found the win you wanted, file it as not tested rather than no effect — otherwise you've buried a good idea under evidence that was never there.

Members only · 4 principles + template + AI mentor

364 more frameworks are waiting.

The Minimum Detectable Effect is Day 1 of 365. One framework every morning for a year — across strategy, sales, negotiation, leadership, and more.

$1 /day

Billed $365/yr · cancel any time

365 frameworks

12 topics · templates · AI mentor

Reserve your place — $365/yr →

One payment a year · the whole library · no auto-renewal.

More Growth frameworks

All 24 growth frameworks →