Sample frameDay 338·Marketing · Measurement

Morning. Something sold yesterday, and a channel took the credit. Those are two separate facts.

Marketing.

The Incrementality Test

5 min read·Apply by lunch

The question

Take the channel you'd defend hardest in a budget meeting — the one whose dashboard glows — and ask what would genuinely have happened last quarter if you'd switched it off across half your territory: would the revenue have dropped, or would those buyers have found another way in? Until you've run that test, you're not measuring a channel — you're paying a toll to stand beside your own front door.

The idea

eBay's economists did something most advertisers never do: they turned the ads off and watched. Thomas Blake and Chris Nosko, working inside the company with Steven Tadelis, an economist at the University of California, Berkeley, stopped buying paid search on the company's own brand keywords — the ads that appear when somebody types 'eBay' into a search engine. Traffic barely moved. Almost everyone who would have clicked the ad clicked the unpaid link sitting underneath it instead. So they ran the harder version, randomising non-brand paid search off across whole US regions and comparing them against the regions left running. Their conclusion, published in Econometrica in 2015, was that returns from paid search were a fraction of what the company's own attribution reports had been claiming, and that whatever real effect existed sat almost entirely with new and infrequent buyers rather than the loyal ones the ads kept taking credit for. Day 48 told you to stop judging brand work with activation's instruments; the holdout is the instrument that settles the argument, because it alone asks what a channel actually caused. The trap is that attribution and incrementality answer different questions. Attribution asks which channel the buyer touched last. Incrementality asks what would have happened without it. Only one of those is a reason to spend money.

The framework

01
Attribution tells you who was standing there. It can't tell you who caused it.
Every attribution model — last click, first click, time-decay, the sophisticated one your platform sells you — divides credit among the touchpoints it can see. None of them asks the counterfactual: would this sale have happened anyway? Channels sitting close to the moment of purchase are structurally flattered by that, because they intercept demand which something else created. Branded search, retargeting and affiliate links are the usual beneficiaries. They aren't necessarily worthless, but a report that can't tell interception from persuasion will rate them highest every single time you run it.
02
Split the map, not the audience.
The cleanest test most businesses can actually run is geographic. Divide your market into regions, hold out a randomly chosen half — or a matched set of cities — and switch the channel off there for long enough that its lag has cleared, usually four to eight weeks. Then compare total revenue between the two groups, per head of population, and ignore clicks entirely. Geography works because it's hard to contaminate: people don't move house to escape your holdout, and the platform can't quietly re-target them. Randomise the split rather than volunteering your weakest regions, or you've built an excuse instead of a test.
03
What would you have to see before you'd actually cut it?
Day 62 made the case for writing down what you expect before the result arrives, and a holdout is where that habit earns its keep, because an uncomfortable number is worth arguing with and somebody will. So name the measure — incremental revenue, or incremental contribution set against the spend — and the threshold at which the channel keeps its budget, halves it, or loses it. Then commit to where the money goes instead. Teams run holdouts, get a result they dislike, and spend three months relitigating the methodology while the spend quietly continues. The pre-committed threshold is the difference between a measurement and a debate.
04
A window with fixed dates and nothing else moving inside it.
The usual way a holdout fails is contamination. A price change, a launch, a rival's promotion or a seasonal cliff landing mid-window will produce a difference that has nothing to do with the channel you switched off. So fix the dates in advance, tell everyone who might change something, and read the whole period rather than the first nervous fortnight. Then diarise the re-run. Incrementality isn't a permanent property of a channel — it shifts as your brand gets stronger, as competitors bid differently, and as more of your demand starts arriving from somewhere else.

Members only · 4 principles + template + AI mentor

364 more frameworks are waiting.

The Incrementality Test is Day 1 of 365. One framework every morning for a year — across strategy, sales, negotiation, leadership, and more.

$1 /day

Billed $365/yr · cancel any time

365 frameworks

12 topics · templates · AI mentor

Reserve your place — $365/yr →

One payment a year · the whole library · no auto-renewal.

More Marketing frameworks

All 28 marketing frameworks →