How Do You Know It Works? Evaluating an AI Legal Workflow

For longer than I’d like to admit, I couldn’t answer a simple question about my own systems.

I changed a prompt. Did it get better?

I had opinions. I was pretty sure the new version wrote better summaries. What I didn’t have was any way to tell “this is better” from “the three examples I happened to look at were better” — or from the far more common and far more dangerous case, “this is better at the thing I was staring at and worse at something I stopped checking six weeks ago.”

Software engineers solved this decades ago. They call it a test suite. Almost nobody using AI for legal work has built one. Here’s how.

It won’t crash. It’ll just get worse.

A crash is fine. It announces itself and somebody fixes it.

What actually hurts you is silent degradation. An extraction workflow that used to catch 97% of deadlines and now catches 89% throws no error, no exception, no alert. It just hands you slightly fewer deadlines, and you don’t notice, because you never knew the true number. That 8% shows up months later as a date nobody calendared.

AI makes this worse than ordinary software in every way. The output isn’t deterministic, so a spot check that passes proves less than it feels like. Changes don’t stay put: edit one instruction and behavior shifts on inputs you never touched. The provider ships a new model and your system changes underneath you. And the output is fluent, so bad output still reads well.

An evaluation set is the only thing that catches this. Nothing substitutes for it, and no amount of “being careful” does the job.

You already have the answer key

Here’s the good news for a law practice: you’re sitting on a pile of documents where you already know the answers.

Closed matters are your golden set. In a closed file you know what the deadlines were, which documents were responsive, what the case was really about, and how it ended. Pull twenty or thirty closed matters, run the workflow on their raw inputs, and compare what it says to what actually happened. That’s a real evaluation set, built from real documents, and about the only cost is the labeling.

How to put it together:

Label once, by hand, and do it carefully. This is the expensive part and you can’t get around it. A day of a lawyer’s time spent labeling thirty examples properly is the best day of engineering you’ll spend, because everything after it runs itself.

Freeze it and version it. The eval set lives under version control like any other code. When you add cases, you cut a new version instead of changing the old one, so today’s score still means something next to March’s.

Load it with the cases that broke you. This is the step people skip, and it’s where most of the value is. Thirty ordinary documents tell you next to nothing, because ordinary documents are easy. Put in the ugly ones:

• The document with no matter number anywhere on it.
• The caption containing a judge whose surname is also a client’s.
• The rule that was renumbered — does the workflow cite the current one?
• The case that was later reversed.
• The scanned fax at 200 dpi with a skew.
• The discovery response that objects and then answers anyway.
• The correspondence that looks urgent and is not, and the one that looks routine and is not.

Every one of those is a real way to fail. A test suite with nothing but the happy path will pass forever while the system rots.

Thirty isn’t a benchmark. It’s also infinitely more than zero. At this size you won’t have statistical power. You’ll still catch every regression that matters, because the ones that matter aren’t subtle 2% shifts. They’re “it stopped extracting service dates entirely.”

Score each job its own way

The biggest mistake is scoring every task the same way. Different tasks fail differently.

Extraction — dates, parties, amounts, citations

Precision and recall against the labeled set, reported separately. Never report a single accuracy number; a system that extracts nothing has terrific precision.

Weight recall far above precision. A date pulled that turns out not to be a deadline costs a lawyer two seconds. A date missed costs a case.

Retrieval

Recall@k, for the reasons I gave in the previous piece. Of the documents you know are relevant, how many show up in the top k? Track it at a few values of k. Recall@10 tells you about the interface. Recall@100 tells you about the pipeline.

Classification — matter matching, inbound triage, document typing

Build a confusion matrix, then go straight to the number that matters: cost-weighted error.

Plain accuracy is close to useless here, because the mistakes are nowhere near equal. In inbound triage, sending a solicitation to the urgent channel costs you a small interruption. Sending a client’s emergency to the suppressed channel costs the client. Those aren’t the same mistake, and a metric that averages them tells you nothing.

So put a price on each kind of mistake. Write the prices down, argue about them, then score against them:

cost = Σ (count of error type i) × (cost weight of error type i)

Writing the weights down is worth as much as the score. You were already having that argument. You were just having it in your head, and badly.

For matter attribution, this gets you an answer people don’t expect: “unattributed” has to be scored as its own outcome, not as a miss. A system that says “I don’t know” is doing its job. A system that guesses wrong just billed the wrong client. If your metric treats those two the same, you’ve told your system to guess.

Citation verification

Two numbers, and I’d put both on a dashboard:

• Verification rate — how many of the citations in a draft survive all four passes.
• Hallucination rate — how many cite authority that doesn’t exist, or doesn’t say what it’s cited for.

The second one should be zero, and you should measure it instead of assuming it. It’s the one metric where the target isn’t “good enough.”

Generation — drafts, summaries, letters

This is the hard one. Three ways to do it, cheapest and least useful first.

Rubric grading. Pick four or five criteria per document type and have a person score each 1–5, every time, against the frozen set. It’s tedious, but it gives you a number you can compare over time.

Pairwise comparison. Show a grader the old output and the new one, unlabeled, in random order, and ask which is better. People are much better at picking between two things than at scoring one thing on a scale, and this catches “it got worse” a lot more sensitively.

LLM-as-judge. It’s cheap, it scales, and it comes with known biases. Position bias: it favors whichever answer came first, so always shuffle the order. Verbosity bias: it likes longer answers whether or not they’re better. Self-preference: it likes output from its own model family. Use it as a smoke alarm, not as the final word. When it disagrees with a person, the person’s right.

Run it every time you touch it

None of this is worth much unless it runs on every change, automatically, and gives you a number you can compare.

Treat your prompts and agent briefs as source code, because that’s what they are. Version them, review them, test them. When you edit one:

1. Run the eval set against the current version. Write down the scores.
2. Make the change.
3. Run it again.
4. Compare — per task and per metric, not one blended number.

The comparison is the whole point. A blended score that went from 0.86 to 0.88 can easily hide extraction getting better while classification fell off a cliff, and if all you look at is the headline, you’ll ship it.

Two more times to run it, beyond your own edits:

When the model changes. A provider shipping a new version is a change to your system that you didn’t make and nobody asked you about. Pin versions where you can, and before you move, run the set.

Every month anyway. Some things drift. A court changes a form, a vendor changes an export format, this year’s documents don’t look like last year’s.

Test the reviewer, too

Almost nobody tracks this one, and I now think it matters most.

Most of these workflows propose and a person approves. So track the acceptance rate, meaning how many proposals go through unchanged, and track the correction rate right next to it.

Here’s why you need both. A rising acceptance rate has two possible causes:

1. The system got better.
2. The human stopped reading.

They look identical in the data, and they’re opposites. The second one is a system that’s quietly gone unsupervised while it keeps piling up an approval record. In a profession where the whole defense of the arrangement is that a lawyer reviewed the work, that’s the worst place you can be.

The only way to tell them apart is to test the reviewer now and then, not just the system. Salt the review queue with a proposal you know is bad — a date that’s deliberately wrong, a matter attribution that’s deliberately off — and see whether anybody catches it. If the planted errors sail through, the review is theater and the acceptance rate means nothing.

It’ll feel like you’re setting a trap for your own staff, or for yourself. Do it anyway. Otherwise you find out the same thing from a client.

Put the price on the scorecard

A workflow that’s 3% more accurate and four times as expensive might be an improvement and might not, and you can’t even have that conversation without the numbers. Log tokens, wall-clock time, and dollar cost per run right next to the quality scores.

Two things show up right away. More steps aren’t always better: a fancier pipeline sometimes does worse, because every extra step is another place to drop information. And speed turns into quality. A workflow slow enough to annoy people is a workflow people go around, and a control everybody goes around isn’t a control.

Start smaller than you think

Everything above is a project. The useful version is an afternoon.

1. Ten examples from closed matters. Label them yourself.
2. One metric per task, picked on purpose. Recall for extraction and retrieval; cost-weighted error for classification.
3. A script that runs the ten and prints a table. No dashboard, no framework, no database.
4. Run it before and after every change, and put the two tables side by side.

That’s a real evaluation harness, and the first time you run it, it’ll tell you things you didn’t know. Everything after that — bigger sets, automated grading, a history of scores — is polishing something that already works.

The one piece you can’t skip is the frozen labeled set. Without it, every claim about whether your system works is a feeling. With it, you’ve got a measurement. And when the output gets signed and filed, that’s not a matter of taste.

General commentary on legal technology and practice management. Not legal advice. All examples are invented; no client matter is described. Evaluation sets built from closed matters contain client information and carry the same confidentiality obligations as the files they came from — see Minn. R. Prof. Conduct 1.6(c).