Retrieval Is the Whole Game

Almost every lousy result I’ve seen from AI in a law office comes back to the same problem, and it isn’t the model.

The model got a good question about the wrong text. Either it never had the document that mattered, or it had ten thousand pages and that document was one paragraph somewhere in the pile, and it answered from what it had. The model takes the blame because that’s where the bad sentence shows up. The mistake happened earlier.

In legal work, retrieval is the whole game. Here’s how to get it right.

Why not just dump the whole file in

Context windows are enormous now, so the obvious move is to skip retrieval: load the whole matter and ask. Sometimes that works. As your default, it fails for three reasons, and only the first is about size.

Where the fact sits matters. The study everybody cites is Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023). The authors tested multi-document question answering and key-value retrieval while moving the relevant information around inside the input. Models did best when the relevant material sat at the beginning or the end, performance degrades significantly when the model has to use information in the middle, and it drops as the input gets longer — even for models built for long context.

Now think about a litigation file. The admission that decides your case is on page 1,847 of a production. That’s the middle. It’s always the middle.

Everything in the window competes. A context isn’t a database you query. The model is looking at all of it at once. Ten thousand pages of routine correspondence don’t sit quietly next to the one letter that matters. They dilute it. Pile irrelevant material into a context and the model gets measurably worse on the material you care about.

And it’s a confidentiality call. Every page you load is a page that went into a conversation. The minimum-necessary instinct that runs everything else in a law office applies here too, and “I loaded the entire client file because it was easier than deciding what mattered” is a lousy answer when somebody asks what you disclosed.

A big context window is room, not a plan. Use it to hold the twelve documents that matter, not to dodge picking which twelve.

Legal documents don’t chunk like prose

Most retrieval tools default to cutting text into fixed windows — 512 tokens, 1,000 characters, with some overlap. That’s built for undifferentiated prose, and it’s flat wrong for legal documents.

Legal documents come with their structure built in, and the structure is where the meaning lives:

Fixed-window chunking cuts paragraph 34 in half and glues the back half to paragraph 35. Now the fragment you retrieve reads like the defendant admitted something they denied. That’s not a little slippage. You just manufactured a fact.

Cut where the document cuts. A complaint splits into numbered paragraphs. Discovery splits into request/response pairs, and the pairing is everything: a response reading “Objection; subject to and without waiving, none” means nothing without the request above it, and a system that stores them apart will confidently hand you half an exchange.

Two more things worth the trouble:

Put the parent context on every chunk. A contract paragraph needs its section heading and the defined terms it uses. Stick a short header on each chunk — document type, party, section — so a retrieved fragment tells you what it is. Retrieval hands back fragments, and a fragment with no provenance is how misattribution starts.

Keep a pointer. Always. Every chunk stores exactly where it came from: document ID, page, paragraph number, Bates range, transcript page:line. That’s what makes citation possible later, and you don’t get to skip it. A system that finds the right text but can’t tell you where it lives has done the easy half.

Why embedding search stumbles on legal text

Vector search — embed the query, embed the chunks, return the nearest neighbors by cosine similarity — is what everybody recommends. On legal text it comes up short, and the reasons aren’t mysterious.

Legal text all looks alike. Every complaint resembles every other complaint. Every discovery objection is built from the same forty phrases. Embeddings pick up topic and style, so when everything shares topic and style, the similarity scores bunch up in a narrow band. The gap between the right chunk and the hundredth-best chunk gets small, and a small gap is noise.

The query is often a token that has to match exactly. “What did they say about the March 14 amendment?” — March 14 and amendment need to match as tokens. So do party names, statute numbers, defined terms, account numbers, and Bates numbers. Dense retrieval is designed to shrug off exact wording, which is backwards when the exact wording is the question. Embeddings will happily hand you the April 2 amendment as a close match.

The little words get lost. “Defendant denies” and “defendant admits” sit right next to each other in embedding space and couldn’t be further apart in meaning. Same with “plaintiff’s expert” versus “defendant’s expert.” Legal meaning turns on exactly the small function words embeddings squash.

Don’t throw dense retrieval out. It earns its keep on paraphrase, where the document says “the parties’ agreement was modified” and you asked about an amendment. Just don’t lean on it alone.

Retrieve in three passes

Pass 1 — Filter on metadata before you search anything. Most legal retrieval questions are filtered questions. “What did the defendant say about the HVAC system in their interrogatory answers” is a metadata filter (matter = X, doc_type = interrogatory_response, party = defendant) plus a search over whatever’s left. Filtering first isn’t a speed trick. You need it to get the right answer. Semantic similarity has no idea “this is the wrong matter,” and a system that can hand you a chunk from another client’s file has a confidentiality problem, not a relevance problem.

That’s why the metadata schema matters as much as the embeddings. At minimum, every chunk carries: matter, document ID and type, date, authoring party, and Bates or page range.

Pass 2 — Hybrid search over what’s left. Run both:

• Lexical (BM25). Exact terms, names, numbers, defined terms.
• Dense (embeddings). Paraphrase and conceptual matches.

Then merge the two ranked lists. Reciprocal rank fusion is the standard way to do it, and for the effort it’s hard to beat — score each document by the sum of 1/(k + rank) across the lists it shows up in, with k a small constant. You don’t have to calibrate scores between the two systems, and that’s the part people usually botch when they try to blend raw scores.

Cast a wide net here — 50 to 100 candidates, not 5.

Pass 3 — Rerank. Take the merged candidates and score each one against the query with a cross-encoder, which reads the query and the chunk together instead of comparing two vectors computed separately. It’s far more accurate and far more expensive, and that’s exactly why it goes last, on 50 candidates instead of 50,000.

Cheap and broad first. Expensive and narrow last.

Recall is the number that counts

This is where legal retrieval parts ways with consumer search, and getting it backwards is the worst design mistake you can make.

In web search, precision wins. The user wants a good answer on top, and nobody cares about the fiftieth result because nobody scrolls. Miss a good page and it costs nothing. There are others.

In a legal matter, missing the document is the entire failure. There’s exactly one email where the CFO writes the sentence that decides the case. Return it at rank 40 and you’ve won. Don’t return it at all and that’s the kind of miss that ends careers. Meanwhile an extra irrelevant result costs a lawyer ten seconds to toss.

That’s a lopsided trade, and it should drive every tuning decision:

• Pull more candidates than feel necessary and let the reranker sort them.
• Never set a similarity threshold that quietly drops results. Return ranked candidates and let a human decide where to stop.
• Stuck between two chunking strategies? Take the one more likely to surface something twice over the one more likely to surface it zero times.
• Report recall@k as the headline metric. Precision is a comfort metric.

And know where this stops. A retrieval system that surfaces 95% of responsive documents is a very good retrieval system, and it’s not a document review. It makes review faster. Anything with a discovery obligation attached still needs a defensible process and a human who’ll sign for it.

Make the model cite what it found

Last piece, and it ties into everything else in this section.

Once retrieval hands back spans with pointers, keep the model on a short leash:

1. Retrieve spans, each carrying its exact source location.
2. Give the model the spans, pointers attached.
3. Tell it every factual assertion must be attributable to a specific span, and that it has to put the pointer right next to the assertion.
4. Check mechanically afterward: every quoted string must appear character-for-character in the span it cites.

That last step is a string comparison, not a judgment call, and it kills a whole category of failure — the made-up quotation — every time, not most of the time. You can tell a model to be careful. It may not listen. A string is either in the source or it isn’t.

What this won’t fix is the harder problem I’ve written about elsewhere: a real span, quoted accurately, cited for something it doesn’t support. That still takes somebody reading the surrounding text and thinking. Good retrieval plus mechanical checks shrinks the problem down to the one spot where you actually need judgment. No tool does better than that.

Build it in this order

If you’re starting from scratch, this order gets you something useful fastest.

1. Metadata and structure-aware chunking first. No embeddings yet. Just parse documents into their native units with full provenance. That alone, with plain keyword search, beats a naive vector store over fixed windows — and if you stop here, you’ve still got something.
2. Add BM25. Cheap, predictable, and it handles the exact-token questions that make up most of what you actually ask.
3. Add dense retrieval and merge the lists. Now you catch paraphrase.
4. Add a reranker when you start seeing good results at rank 30.
5. Add citation-anchored generation with mechanical quote verification.

Each step pays off on its own, and each one shows you where the next one breaks. Build the whole pipeline before you test any of it and you’ve got a system that’s wrong somewhere and no way to find out where. That’s the next piece.

Sources

• Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang, Lost in the Middle: How Language Models Use Long Contexts (2023)

General commentary on legal technology. Not legal advice. Nothing here describes a client matter; example documents are invented. Any system touching client files carries the confidentiality obligations discussed elsewhere in this section — see Minn. R. Prof. Conduct 1.6(c) and ABA Formal Opinion 512.