Sample frameDay 139·AI Fluency · Testing

Good morning. Everyone on your team has a favourite model. Not one of them can prove it.

AI Fluency.

The Held-Out Ten

5 min read·Apply by lunch

The question

When you last rewrote a prompt or moved to a newer model, what did you compare the new version against? If it was the two or three outputs you happened to glance at afterwards, you haven't measured anything. You've changed your mind and called it a test.

The idea

Ron Kohavi ran the experimentation platform at Microsoft and, before that, data mining at Amazon; with Diane Tang of Google and Ya Xu of LinkedIn he wrote Trustworthy Online Controlled Experiments (2020), the standard text on testing changes at scale. Its most useful number is also its most deflating: of the ideas Microsoft put through controlled experiments, only about a third improved the metric they were built to improve. A third changed nothing. A third made things worse. These weren't careless ideas. They were the surviving proposals of experienced product people who had argued for them in rooms full of other experienced people. The lesson carries straight into AI work, where you change a prompt, a model or a tool almost weekly and judge the change by whether the last few outputs felt better. The trap is that you're both the person who chose the change and the person marking it, working from memory, on a sample of two. Without a fixed set of cases and a known-good answer for each, you can't tell an improvement from a good mood.

The framework

01
The set starts with the five you already have.
Day 99 had you re-run five finished jobs cold to find the edge of the model's competence. Those five are the beginning of this set. Go into the archive and add five more where you know what correct looks like: the summary you wrote and stood behind, the classification someone checked. Real inputs, messy formatting included. Spread them on purpose — three routine, four typical, two genuinely hard, one that has burned you before — because a set built only from easy cases will report that every option is excellent. The temptation is to invent ten clean examples this afternoon rather than dig up ugly ones from last quarter. Invented cases test the tool against your imagination, which is the judge you were trying to replace.
02
Write the answer key before you see a single output.
For each case, set down what a correct answer contains: the facts that must appear, the figure that must be right, the format, the claim it must not make. Do this while you still have nothing in front of you, because grading after the fact isn't grading — you'll mark the output correct against criteria you invented in the previous ten seconds. Score each case pass or fail, and refuse half marks. A partial credit scheme is where the argument you were trying to settle walks back in through the side door.
03
The whole set, every time, and the number written down.
New model, rewritten prompt, impressive demo: run all ten, not the two you remember. Log the score and the date in one file — nine of ten on 4 March, seven of ten after the rewrite — and keep the failures themselves, because which case broke tells you more than the total. Most people stop here, having confirmed what they already believed. The discipline is running the set on the changes you expect to like, so that the one time the number disagrees with you, it's sitting there in the same format as every other run.
04
Retire the cases that go quiet; add the ones that hurt.
A held-out set decays. Cases every option now passes have stopped carrying information, and this year's work isn't last year's work. So run it as a living file: each time something goes wrong in real use — the invented figure, the misread clause, the tone that went out under your name — that failure becomes a new case with a known-good answer attached. Cap the set at ten or twelve so it stays cheap to run. Sixty cases is a project. Ten is twenty minutes on a Tuesday.

Members only · 4 principles + template + AI mentor

364 more frameworks are waiting.

The Held-Out Ten is Day 1 of 365. One framework every morning for a year — across strategy, sales, negotiation, leadership, and more.

$1 /day

Billed $365/yr · cancel any time

365 frameworks

12 topics · templates · AI mentor

Reserve your place — $365/yr →

One payment a year · the whole library · no auto-renewal.

More AI Fluency frameworks

All 20 ai fluency frameworks →