Towards Reliable Retrieval in RAG Systems for Large Legal Datasets

Markus Reuter, Tobias Lingenberg, Ruta Liepina, Francesca Lagioia, Marco Lippi, Giovanni Sartor, Andrea Passerini, Burcu Sayin

Introduction

Retrieval-Augmented Generation (RAG) has become the standard architecture for grounding Large Language Models (LLMs) in external knowledge. However, in specialized domains like law, standard RAG pipelines often falter. The primary challenge is not the generation phase, but the retrieval phase, where highly similar documents—such as boilerplate contracts—confuse the system. This paper introduces Summary-Augmented Chunking (SAC), a method designed to preserve global document context within local text chunks, significantly improving retrieval accuracy.

The Problem: Document-Level Retrieval Mismatch (DRM)

Legal datasets are characterized by high lexical redundancy. Thousands of documents, such as Non-Disclosure Agreements (NDAs) or Merger Agreements, may share nearly identical phrasing, differing only in critical details like party names or dates.

When these documents are split into smaller chunks for vector indexing, they lose their "global" identity. A retriever might find a chunk that semantically matches a user's query but belongs to the wrong document. The authors define this failure as Document-Level Retrieval Mismatch (DRM). In legal contexts, retrieving a clause from the wrong contract is as detrimental as a complete hallucination.

Methodology: Summary-Augmented Chunking (SAC)

To address DRM, the authors propose Summary-Augmented Chunking (SAC). Unlike complex architectural shifts, SAC is a lightweight pre-retrieval enhancement.

Overview of the LegalBench-RAG metrics and the Summary-Augmented Chunking (SAC) workflow.

The process involves:

  1. Document Summarization: Using a model like GPT-4o-mini to generate a concise (~150 character) summary of the entire document.
  2. Recursive Chunking: Splitting the document into standard chunks (e.g., 500 characters).
  3. Augmentation: Prepending the document-level summary to every chunk derived from that document.
  4. Indexing: Embedding these "augmented" chunks into a vector database.

The summary acts as a unique "fingerprint," ensuring that even if the chunk text is boilerplate, the embedded vector contains enough context to distinguish it from similar chunks in other documents.

Experimental Setup

The authors evaluated SAC using LegalBench-RAG, a benchmark designed to isolate retrieval performance. The benchmark utilizes four diverse legal datasets:

  • CUAD: General commercial contracts.
  • MAUD: Merger and acquisition agreements.
  • ContractNLI: Non-disclosure agreements.
  • PrivacyQA: Mobile application privacy policies.

Two summarization strategies were compared: a generic prompt for concise summarization and an expert-guided prompt designed to extract specific legal variables (jurisdiction, parties, etc.).

Key Results and Discussion

The implementation of SAC yielded several significant insights:

1. Significant Reduction in DRM

SAC effectively halved the DRM rate compared to standard RAG baselines. By providing the retriever with document-level context, the system was far more likely to select chunks from the correct source document.

2. The "Generic vs. Expert" Paradox

Surprisingly, generic summaries outperformed expert-crafted legal summaries. While expert summaries were more structured, they appeared to "overfit" to specific legal features, making them less flexible for a variety of user queries. Generic summaries provided a broader semantic anchor that better aligned with natural language queries.

3. Improved Precision and Recall

By successfully navigating to the correct document, the system's ability to pinpoint the exact relevant text snippet improved.

Comparison of text-level precision and recall across different retrieval methods. SAC consistently outperforms the baseline.

As shown in the performance plots, SAC (using generic summaries) consistently maintains higher precision and recall across various top-k settings compared to both the baseline and the expert-guided version.

Conclusion

Summary-Augmented Chunking offers a simple yet powerful solution to the context-loss problem in legal RAG systems. By prepending a document-level "fingerprint" to every chunk, SAC mitigates the risks posed by lexical redundancy and boilerplate text. Future work will explore hierarchical summarization and the integration of SAC into end-to-end generation pipelines to measure its impact on final answer quality.

Summary updated

BibTeX
@misc{reuter2025towards,
  author = {Markus Reuter and Tobias Lingenberg and Ruta Liepina and Francesca Lagioia and Marco Lippi and Giovanni Sartor and Andrea Passerini and Burcu Sayin},
  title = {{Towards Reliable Retrieval in RAG Systems for Large Legal Datasets}},
  year = {2025},
  eprint = {2510.06999},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.06999},
}
Figure