Growth.

The Overall Evaluation Criterion

5 min read·Apply by lunch

The question

Could you write down, before this week's result lands, the one number your current test wins or loses on, the rate at which you'd trade that number against the things the change might damage, and the two outcomes that would stop it shipping even on a win? If those three are still unwritten when the numbers arrive, the criterion gets picked by whoever most wants the change to have worked.

The idea

In August 2012 six Microsoft researchers — Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu — took a paper to the KDD conference in Beijing, subtitled Five Puzzling Outcomes Explained. The first puzzle is a bug. A ranking fault in a Bing experiment served users visibly poor search results, and two of the company's headline numbers rose: distinct queries per user by over ten per cent, revenue per user by over thirty. Frustrated people search again, and click ads that suddenly look relevant by comparison. Judged on either number, the right call was to ship the damage. The instrument that stops you is the Overall Evaluation Criterion, which Kohavi set out at book length eight years later with Diane Tang and Ya Xu in Trustworthy Online Controlled Experiments. It's one quantitative measure of what a test is for, settled before the test runs: the metrics the change could trade against each other, combined into a single number, sitting beside a short list of things it isn't allowed to break. Day 23's North Star belongs to the company and moves rarely. This one belongs to the test, and you write it the week you write the hypothesis.

The framework

01
One number, because a test can't be allowed to win twice.
A criterion is singular by construction. Two metrics let a change win on whichever one happened to move, and that choice gets made after the result is visible, usually by the person who proposed the change. So the combining has to happen first. If this could lift conversion and lift refunds, decide this morning what a point of conversion is worth in refunds, and carry that exchange rate into the criterion itself. The rate will be crude. Crude and agreed beats exact and improvised, because the version invented on Friday is always the version that ships the change.
02
A broken search engine improved two of Bing's headline numbers.
That's the check to run on any criterion before you trust it: how would you beat this by doing something you'd be embarrassed to explain? Thomas Crook and colleagues put it first among seven pitfalls at the same conference in 2009 — choosing a criterion you can beat by doing something clearly wrong for the business. Degrade the results and queries rise. Add a step and time on page rises. Mail everyone twice and sessions rise. If the obviously destructive version wins, you haven't written a criterion, you've written an instruction, and somebody eventually follows it.
03
Break the long-term goal into its terms, then see which one a test can move.
Bing's long-term goal was query share, so the team split monthly distinct queries into three multiplied terms: queries per session, sessions per user, and users per month. Each behaves differently inside an experiment. Users per month is fixed by the split itself, so it can't sit in the criterion at all. Queries per session should fall rather than rise, provided the task still gets finished. Sessions per user is the term a satisfied customer produces, and that became the component they optimised. Do the same arithmetic on your own long-term number and most candidates disqualify themselves before anyone argues.
04
Guardrails aren't a second scoreboard.
The list beside the criterion is short, named in advance, and made of things a win isn't allowed to cost: unsubscribes, refunds, support contacts, load time, the churn rate of the accounts you already have. Each one is a stop, not a tiebreak — you don't average a guardrail against the criterion, you check whether it broke. Bing handled revenue this way, optimised inside an agreed allowance of average pixels per query rather than maximised. Keep the list to about three. A dozen guardrails is a dashboard, and a dashboard always finds somebody a reason to ship.

Members only · 4 principles + template + AI mentor

364 more frameworks are waiting.

The Overall Evaluation Criterion is Day 1 of 365. One framework every morning for a year — across strategy, sales, negotiation, leadership, and more.

$1 /day

Billed $365/yr · cancel any time

365 frameworks

12 topics · templates · AI mentor

Reserve your place — $365/yr →

One payment a year · the whole library · no auto-renewal.

More Growth frameworks

All 24 growth frameworks →