Here is a question I could not answer about my own systems for longer than I would like to admit.
I changed a prompt. Did it get better?
I had opinions. I had the strong impression that the new version was producing better summaries. What I did not have was any way to distinguish “this is better” from “the three examples I happened to look at were better” — or from the far more common and far more dangerous case, “this is better at the thing I was staring at and worse at something I stopped checking six weeks ago.”
Software engineering solved this problem decades ago and called it a test suite. Almost nobody applying AI to legal work has built one. This is how.
The failure you are actually defending against
Not a crash. Crashes are fine — they announce themselves and somebody fixes them.
The thing to defend against is silent degradation. An extraction workflow that used to catch 97% of deadlines and now catches 89% produces no error, no exception, no alert. It produces slightly fewer deadlines, and you do not notice, because you never knew the true number. The 8% shows up months later as a date nobody calendared.
Every property of AI systems makes this worse than it would be in ordinary software. Output is non-deterministic, so a spot check that passes proves less than it feels like it proves. Changes are non-local — editing one instruction shifts behavior on unrelated inputs. Models change underneath you when a provider ships a new version. And the output is fluent, which means degraded output still reads well.
An evaluation set is the only instrument that detects this. There is no substitute and there is no version of “being careful” that does the job.
Building ground truth
The good news for a law practice: you are sitting on a corpus with known answers.
Closed matters are your golden set. For a closed file you know what the deadlines were, which documents were responsive, what the case was actually about, how it resolved. Take twenty or thirty closed matters, run the workflow against their raw inputs, and compare to what actually happened. That is a real evaluation set built from real documents at essentially no cost beyond the labeling.
Rules for assembling it:
Label once, carefully, by hand. This is the expensive part and there is no way around it. A day of a lawyer’s time producing a properly labeled set of thirty examples is the best day of engineering you will spend, because everything after it is automatic.
Freeze it and version it. The eval set is an artifact under version control. When you add cases, you add a new version rather than mutating the old one, so a score today is comparable to a score in March.
Include the adversarial cases deliberately. This is the part people skip and it is where the value concentrates. A set of thirty typical documents tells you almost nothing, because typical documents are easy. Load it with the cases that broke you:
- The document with no matter number anywhere on it.
- The caption containing a judge whose surname is also a client’s.
- The rule that was renumbered — does the workflow cite the current one?
- The case that was later reversed.
- The scanned fax at 200 dpi with a skew.
- The discovery response that objects and then answers anyway.
- The correspondence that looks urgent and is not, and the one that looks routine and is not.
Each of those represents a real failure mode. A test suite that contains only the happy path is a suite that will pass forever while the system rots.
Thirty is not a benchmark, and thirty is infinitely more than zero. You will not have statistical power at this size. You will still catch every regression that matters, because the regressions that matter are not subtle 2% shifts — they are “it stopped extracting service dates entirely.”
Metrics, by task type
The single biggest mistake is scoring every task the same way. Different tasks fail differently.
Extraction — dates, parties, amounts, citations
Precision and recall against the labeled set, reported separately. Never report only an aggregate accuracy; a system that extracts nothing has excellent precision.
Weight recall far above precision. A date extracted that turns out not to be a deadline costs a lawyer two seconds. A date missed costs a case.
Retrieval
Recall@k, for the reasons I set out in the previous piece. What fraction of the known-relevant documents appear in the top k? Track it at several values of k — recall@10 tells you about the interface, recall@100 tells you about the pipeline.
Classification — matter matching, inbound triage, document typing
A confusion matrix, and then the number that actually matters: cost-weighted error.
Accuracy is close to meaningless here because the errors are wildly asymmetric. In inbound triage, routing a solicitation to the urgent channel costs a small interruption. Routing a client emergency to the suppressed channel costs the client. Those are not the same error and no metric that averages them is telling you anything.
So assign explicit costs. Write them down, argue about them, and then score against them:
cost = Σ (count of error type i) × (cost weight of error type i)The discipline of writing the weights down is worth as much as the score. It forces an argument you were having implicitly and badly.
For matter attribution specifically, this framework produces a result people find counterintuitive: “unattributed” must be scored as a distinct outcome, not as a miss. A system that says “I don’t know” is behaving correctly. A system that guesses wrong has produced a bill to the wrong client. If your metric treats those identically, you have instructed your system to guess.
Citation verification
Two numbers, and I would put both on a dashboard:
- Verification rate — what fraction of citations in a draft survive all four passes.
- Hallucination rate — what fraction of citations refer to authority that does not exist, or does not say what it is cited for.
The second should be zero and you should measure it rather than assume it. This is the one metric where the target is not “good enough.”
Generation — drafts, summaries, letters
The hard one. Three approaches, in ascending order of cost and usefulness.
Rubric grading. Define four or five criteria per document type, score each 1–5, by hand, against the frozen set. Tedious, but it produces a comparable number over time.
Pairwise comparison. Show a grader the old output and the new output, unlabeled and in random order, and ask which is better. People are far more reliable at ranking two things than at scoring one thing absolutely, and this catches “it got worse” much more sensitively.
LLM-as-judge, with its failure modes acknowledged. It is cheap and it scales, and it has known biases: position bias (a preference for whichever answer came first — always randomize order), verbosity bias (a preference for longer answers regardless of quality), and self-preference (a preference for output from the same model family). It is useful as a regression alarm, not as an authority. When it disagrees with a human, the human is right.
Regression testing
The point of all of the above is that it runs on every change, automatically, and produces a comparable number.
Treat your prompts and agent briefs as source code, because that is what they are. They are versioned, they are reviewed, and they have tests. When you edit one:
- Run the eval set against the current version. Record the scores.
- Make the change.
- Run it again.
- Compare — per task and per metric, not in aggregate.
The comparison is the entire product. An aggregate score that went from 0.86 to 0.88 can easily conceal extraction improving while classification fell off a cliff, and if you only look at the headline you will ship that.
Two additional triggers beyond your own edits:
Re-run when the model changes. A provider shipping a new version is a change to your system that you did not make and were not consulted about. Pin versions where you can, and when you move, run the set first.
Re-run periodically anyway. Monthly is plenty. Some things drift — a court changes a form, a vendor changes an export format, the documents arriving this year are not shaped like last year’s.
Measure the human, too
The metric almost nobody tracks and the one I now consider most important.
Most of these workflows propose and a human approves. So track the acceptance rate — what fraction of proposals go through unchanged — and track the correction rate alongside it.
Here is why this pairing matters. A rising acceptance rate has two possible causes:
- The system got better.
- The human stopped reading.
These look identical in the data and they are opposite conditions. The second is a system that has quietly become unsupervised while continuing to generate an approval record — which, in a professional context where the whole defense of the arrangement is that a lawyer reviewed it, is the worst possible state.
Distinguishing them requires occasionally testing the reviewer, not just the system. Salt the review queue with a known-bad proposal — a date that is deliberately wrong, a matter attribution that is deliberately off — and see whether it gets caught. If seeded errors sail through, the review is theater and the acceptance rate is meaningless.
That will feel adversarial toward your own staff, or toward yourself. Do it anyway. The alternative is discovering the same fact from a client.
Cost and latency belong in the eval
A workflow that is 3% more accurate and four times as expensive may or may not be an improvement, and you cannot have that conversation without the numbers. Record tokens, wall-clock time, and dollar cost per run alongside the quality scores.
Two effects show up immediately and are worth anticipating. Quality and cost are not monotonic — a more elaborate pipeline sometimes performs worse, because more steps means more places to lose information. And latency has a quality effect of its own: a workflow slow enough to be annoying is a workflow people route around, and a bypassed control is not a control.
Start smaller than you think
The complete version above is a project. The useful version is an afternoon.
- Ten examples from closed matters. Label them by hand.
- One metric per task, chosen deliberately. Recall for extraction and retrieval; cost-weighted error for classification.
- A script that runs the ten and prints a table. No dashboard, no framework, no storage layer.
- Run it before and after every change, and paste the two tables side by side.
That is a real evaluation harness and it will immediately tell you things you did not know. Everything past it — bigger sets, automated grading, tracked history — is refinement of something that is already working.
The one thing that is not optional is the frozen labeled set. Without it, every claim about whether your system works is a feeling. With it, they are measurements. In a profession where the output gets signed and filed, that distinction is not a matter of engineering taste.
General commentary on legal technology and practice management. Not legal advice. All examples are invented; no client matter is described. Evaluation sets built from closed matters contain client information and carry the same confidentiality obligations as the files they came from — see Minn. R. Prof. Conduct 1.6(c).