AI Fluency.

The framework

1. 01

   Your ten cases already exist. Add the ones this decision exposes.

   Day 139 had you build the held-out set: ten pieces of real work with a known-good answer attached to each. That's the spine of the bake-off, and you don't rebuild it every time somebody books a demo. What a purchase adds is the handful of cases the purchase itself exposes — the ugly jobs a new system would meet in its first month, and the two or three the vendor's pitch implies they're strongest at. Vendors will happily supply an evaluation set and it'll be an excellent one; it'll also be the set their system was tuned against. Your cases, or the exercise measures nothing you care about.

2. 02

   Five columns, scored on one day, from the same cases.

   Accuracy, marked against the key you already wrote. Cost per case at your real monthly volume, not the headline price per million tokens. Latency at the moment your team would actually be sitting waiting. Failure mode, which earns its own frame below. And the contract terms, which decide whether the other four still hold in a year. Run both candidates and your current process, the control, through all five on one day with identical inputs. A comparison assembled across three weeks measures the weather rather than the systems.

3. 03

   Read how it fails, not only how often.

   Two systems on the same score aren't equivalent. One fails loudly: it refuses, flags low confidence, produces something visibly wrong that any reviewer catches in a second. The other fails quietly — the same shape of answer it always gives, with one figure changed. At the same score the second is the more expensive purchase, because your review process was built for humans, and humans hesitate when they're unsure. Score the failure mode explicitly and weight it above a couple of accuracy points. You're buying a review burden as much as an output.

4. 04

   What happens to your work the day you leave?

   Get five answers in writing before you look at a single score: is your input used to train the model, how long is it retained, who inside the vendor can see it, where does it physically sit, and what happens to it the day you go. Then ask the exit question plainly. If you cancelled next month, what could you take with you, in what format, and how long would that take? A system that wins on every metric and holds your work hostage hasn't won anything. Put those answers on the scorecard as a column of their own, weighted like the other four.