Retrieval Is the Whole Game

Almost every disappointing result I have seen from AI in a law office traces back to the same root cause, and it is not the model.

The model was asked a good question about the wrong text. Either it did not have the document that mattered, or it had ten thousand pages in which that document was one paragraph, and it answered from what it had. The generation step gets the blame because that is where the wrong sentence appears. The failure happened upstream.

In legal work, retrieval is the whole game. This is the technical piece on how to do it properly.

Why not just put everything in the context window

The obvious move, now that context windows are enormous, is to skip retrieval entirely: load the whole matter and ask. Sometimes this works. As a default architecture it fails for three reasons, and only the first is about size.

Position effects are real and well documented. The canonical study is Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023), which measured multi-document question answering and key-value retrieval as a function of where the relevant information sat. The finding: performance is highest when the relevant material is at the beginning or the end of the input, and degrades significantly when the model has to use information in the middle — and performance drops as the input grows longer, including for models explicitly built for long context.

Now map that onto a litigation file. The dispositive admission is on page 1,847 of a production. That is the middle. It is always the middle.

Everything in the window competes. A context is not a database that gets queried; it is material the model is attending to all at once. Ten thousand pages of routine correspondence do not sit inertly beside the one important letter — they dilute it. Adding irrelevant material to a context measurably degrades performance on the relevant material.

And it is a confidentiality decision. Every page you load is a page that entered a conversation. The minimum-necessary instinct that governs everything else in a law office applies here too, and “I loaded the entire client file because it was easier than deciding what mattered” is a poor answer to the question of what you disclosed.

Long context is a capacity, not a strategy. Use it to hold the twelve documents that matter, not to avoid deciding which twelve.

Chunking legal documents is not chunking prose

The default in most retrieval tooling is to split text into fixed windows — 512 tokens, 1,000 characters, with some overlap. This is designed for undifferentiated prose and it is actively wrong for legal documents.

Legal documents already have structure, and that structure is the unit of meaning:

Fixed-window chunking cuts paragraph 34 in half and glues the back of it to paragraph 35. The retrieved fragment then reads as though the defendant admitted something they denied. That is not a subtle degradation — it is a manufactured fact.

Chunk on the document’s own boundaries. A complaint chunks into numbered paragraphs. Discovery chunks into request/response pairs, and the pairing is essential: a response reading “Objection; subject to and without waiving, none” is meaningless without the request above it, and a system that stores them separately will confidently retrieve half of an exchange.

Two refinements worth the effort:

Carry the parent context on every chunk. A paragraph from a contract needs its section heading and the defined terms it uses. Prepend a short header to each chunk — document type, party, section — so a retrieved fragment is self-describing. Retrieval returns fragments; fragments without provenance are how misattribution starts.

Keep a pointer, always. Every chunk stores exactly where it came from: document ID, page, paragraph number, Bates range, transcript page:line. This is what makes citation possible later, and it is not optional. A retrieval system that can find the right text but cannot say where it lives has solved the easy half.

Why embedding search underperforms on legal text

Vector search — embed the query, embed the chunks, return nearest neighbors by cosine similarity — is the default recommendation everywhere. On legal corpora it underperforms, and the reasons are specific rather than mysterious.

Legal text is pathologically self-similar. Every complaint resembles every other complaint. Every discovery objection is built from the same forty phrases. Embeddings capture topical and stylistic similarity, so in a corpus where everything shares topic and style, the similarity scores compress into a narrow band. The gap between the right chunk and the hundredth-best chunk gets small, and small gaps are noise.

The query is often a token that must match exactly. “What did they say about the March 14 amendment?” — March 14 and amendment need to match as tokens. So do party names, statute numbers, defined terms, account numbers, and Bates numbers. Dense retrieval is designed to be robust to exact wording, which is precisely the wrong property when the exact wording is the query. Embeddings will happily return the April 2 amendment as similar.

Negation and direction are poorly represented. “Defendant denies” and “defendant admits” are minimally distant in embedding space and maximally distant in meaning. Same for “plaintiff’s expert” versus “defendant’s expert.” Legal meaning turns on exactly the small function words that embeddings compress.

The fix is not to abandon dense retrieval — it earns its keep on paraphrase, where the document says “the parties’ agreement was modified” and you asked about an amendment. The fix is not to rely on it alone.

Retrieve in three stages

Stage 1 — Filter on metadata, before any search. Most legal retrieval questions are filtered questions. “What did the defendant say about the HVAC system in their interrogatory answers” is a metadata filter (matter = X, doc_type = interrogatory_response, party = defendant) plus a search over the survivors. Doing the filter first is not an optimization; it is a correctness requirement. Semantic similarity has no concept of “this is the wrong matter,” and a system that can return a chunk from another client’s file has a confidentiality defect, not a relevance defect.

This is why the metadata schema matters as much as the embeddings. At minimum, every chunk carries: matter, document ID and type, date, authoring party, and Bates or page range.

Stage 2 — Hybrid search over what survives. Run both:

• Lexical (BM25). Exact terms, names, numbers, defined terms.
• Dense (embeddings). Paraphrase and conceptual matches.

Then fuse the two ranked lists. Reciprocal rank fusion is the standard approach and is hard to beat for the effort — score each document by the sum of 1/(k + rank) across the lists it appears in, with k a small constant. It needs no score calibration between the two systems, which is the part that usually goes wrong when people try to blend raw scores.

Retrieve generously here — 50 to 100 candidates, not 5.

Stage 3 — Rerank. Take the fused candidates and score each against the query with a cross-encoder, which reads query and chunk together rather than comparing two independently-computed vectors. This is far more accurate and far more expensive, which is exactly why it goes last, on 50 candidates rather than 50,000.

The shape of the whole thing: cheap and broad, then expensive and narrow.

Recall is the metric, not precision

This is where legal retrieval diverges most sharply from consumer search, and getting it backwards is the most consequential design error available.

In web search, precision dominates. The user wants a good answer at the top; the fiftieth result is irrelevant because nobody scrolls. Missing a good page costs nothing — there are others.

In a legal matter, missing the document is the entire failure. There is exactly one email in which the CFO writes the sentence that decides the case. Returning it at rank 40 is a total success; failing to return it at all is the thing that ends careers. Meanwhile the cost of an extra irrelevant result is that a lawyer spends ten seconds discarding it.

The asymmetry is enormous and it should drive every tuning decision:

• Retrieve more candidates than feel necessary and let the reranker sort them.
• Never set a similarity threshold that silently drops results. Return ranked candidates; let a human decide where to stop.
• When in doubt between two chunking strategies, take the one that is more likely to surface something twice than the one more likely to surface it zero times.
• Report recall@k as the headline metric. Precision is a comfort metric.

And state the limit honestly: a retrieval system that surfaces 95% of responsive documents is a very good retrieval system and is not a document review. It is a tool that makes review faster. Anything with a discovery obligation attached to it still needs a defensible process and a human who will sign for it.

Citation-anchored generation

The final piece, and the one that connects to everything else in this section.

Once retrieval returns spans with pointers, the generation step should be constrained to them:

1. Retrieve spans, each carrying its exact source location.
2. Give the model the spans, with pointers attached.
3. Instruct it that every factual assertion must be attributable to a specific span, and that it must emit the pointer alongside the assertion.
4. Verify mechanically afterward: every quoted string must appear character-for-character in the span it cites.

That last step is a string comparison, not a judgment call, and it eliminates an entire category of failure — the fabricated quotation — deterministically rather than probabilistically. A model can be instructed to be careful and may not be. A string either appears in the source or it does not.

What this does not solve is the harder problem I have written about elsewhere: a real span, quoted accurately, cited for a proposition it does not support. That still requires somebody to read the surrounding text and think. Retrieval and mechanical verification narrow the failure surface to the one place where judgment is genuinely required, which is the most any tooling can do.

The build order

If you are starting, this is the order that produces useful results earliest.

1. Metadata and structure-aware chunking first. No embeddings at all yet. Just parse documents into their native units with full provenance. This alone, with plain keyword search, is more useful than a naive vector store over fixed windows — and if you stop here you still have something.
2. Add BM25. Cheap, deterministic, and it handles the exact-token queries that are most of what you actually ask.
3. Add dense retrieval and fuse. Now you catch paraphrase.
4. Add a reranker when you notice good results appearing at rank 30.
5. Add citation-anchored generation with mechanical quote verification.

The order matters because each stage is independently useful and each one makes the next one’s failures visible. Building the whole pipeline before evaluating any of it produces a system that is wrong somewhere and gives you no way to find out where — which is the subject of the next piece.

Sources

• Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang, Lost in the Middle: How Language Models Use Long Contexts (2023)

General commentary on legal technology. Not legal advice. Nothing here describes a client matter; example documents are invented. Any system touching client files carries the confidentiality obligations discussed elsewhere in this section — see Minn. R. Prof. Conduct 1.6(c) and ABA Formal Opinion 512.