Evaluation

The evaluation set is written before the feature is built. It is a list of cases, each with an input and a statement of what an acceptable response looks like, and it is authored first for the same reason the depth gate was: a set assembled afterward gets assembled from the behavior the system already has, which guarantees it passes and measures nothing.

What counts as wrong has to be defined for the specific domain, because the word covers unlike things. A summary that omits a material term, a classifier that routes a complaint to the wrong queue, an extractor that reads the renewal date instead of the signature date — these carry different costs and demand different responses, and a single accuracy figure averages them into a number that conceals which one is happening.

Cases are drawn from real inputs, including the ones nobody would choose for a demonstration: the scanned fax, the message with a signature block and a legal footer attached, the customer who asked three questions in one sentence and contradicted themselves in the third. A set built from clean examples measures a system that will never be deployed.

Adversarial and boundary cases belong in the same set instead of in a separate exercise done later if there is time. The empty input, the input in another language, the request that should be declined, the document containing text addressed to the system — each has a correct behavior, and a correct behavior that has not been written down is one nobody can verify.

Grading is mechanical where it can be. Cases with an exact answer are checked by comparison, which is cheap enough to run constantly. Where judgement is genuinely required and a model performs the grading, that grader is itself validated against human judgements on a sample before its verdicts are trusted, because an unvalidated automatic grader is a second unmeasured system layered on the first.

An evaluation set is an asset with a running cost, and it is sized with that in mind. A set large enough to be statistically comfortable and slow enough that nobody wants to run it will end up run on no changes at all, which is a worse position than a smaller set taken seriously. The workable arrangement is usually a fast subset on every deploy and the full set on a schedule.

The evaluation runs on a schedule rather than once before launch, and this is the property most often skipped. A language model's behavior can change with no change whatsoever to the code: a vendor ships an update, retires a version, adjusts a default, or alters a safety boundary. Nothing in a normal deployment pipeline observes any of that. A standing evaluation is the only mechanism that will notice, and without one the first report of a regression comes from a customer.

Model versions are pinned wherever the vendor permits it, so that a change becomes something the business schedules instead of something it receives. Pinning is not a substitute for evaluation and is not treated as one — deprecation windows close, and a pinned version is a deferral with a date on it, not a decision that holds.

Results are reported per class of case instead of as one headline figure. An overall number that improves while the class covering regulated language gets worse is a number that hides the only fact anyone needed, and the aggregate will look reassuring for exactly as long as it takes for the regression to reach someone who cares about that class.

Any case that failed once and was fixed stays in the set permanently. This is ordinary regression practice and it is skipped in this domain more than anywhere else, because the fix is usually an edit to a prompt and a prompt edit does not feel like a code change. It is one, and it can undo an earlier fix as easily as any refactor.

Prompts are therefore versioned, reviewed and deployed like code. A prompt altered directly in a vendor console is an untracked production change with no author, no diff and no way back, and it is the single most common source of behavior nobody can account for later.

After launch, live traffic is sampled and fed back into the set, because the written cases capture what was anticipated and real inputs diverge from that within a fortnight. The sample is what converts an unexpected failure into a permanent case, not an anecdote somebody remembers for a while.

The threshold for shipping is agreed before the measuring begins. A failure rate accepted in advance is a decision; the identical rate presented once the work is finished is a negotiation held from a position where stopping has already become expensive. Fixing the number first also forces the more useful conversation, which is what the business intends to do about the cases that fail rather than whether any exist.

The comparison is against the process being replaced, not against perfection. The manual alternative has an error rate as well, it is almost never measured, and so a system gets rejected for failures that are newly visible while the process it would have replaced goes on making unexamined ones. Establishing that baseline is often the hardest part of the engagement and occasionally the least comfortable, because it means measuring how frequently people currently get it wrong.

None of this makes a system correct, and the work is not sold as though it did. It makes the failure rate a known quantity, attached to a date and a case set, that a person can look at and accept or reject. That is a lower bar than correctness. It is also the only bar actually on offer, and a system whose failure rate is unknown has not cleared it.