LegalSearchLM: Rethinking Legal Case Retrieval as Legal Elements Generation

Chaeeun Kim, Jinu Lee, Wonseok HwangEMNLP 2025

TL;DR

This paper introduces LEGAR BENCH, the first large-scale Korean legal case retrieval benchmark, and LEGALSEARCHLM, a novel generative retrieval model. LEGALSEARCHLM addresses limitations of existing methods by generating key legal elements from a query case to directly retrieve relevant documents, achieving state-of-the-art performance and demonstrating strong generalization.

What Problem Does This Paper Address?

Legal Case Retrieval (LCR) is crucial for legal professionals to find relevant precedents. However, existing LCR research faces two main challenges:

  1. Limited Evaluation Scales: Most benchmarks use small corpora (100-55K cases) and narrow query types, not reflecting real-world complexity.
  2. Inadequate Representation: Current models rely on embedding-based similarity or lexical matching, which struggle to capture the rich semantics of legal texts, leading to irrelevant matches or information loss.

Methods

The authors propose LEGALSEARCHLM, a generative retrieval model that shifts LCR from sequence-to-sequence matching to generating key legal elements.

  • Legal Element Generation: LEGALSEARCHLM directly generates legal elements that should be present in target documents. This is achieved through a generative approach, treating retrieval as a language modeling task.
  • First-Token-Aware Generation: To prevent early decoding decisions from steering the model towards irrelevant paths (e.g., dates, locations), they employ a strategy that prioritizes legally informative tokens from legal elements.
  • Self-Supervised Fine-Tuning (SSFT): Given the cost of legal annotation, they use a self-supervised approach. The model is trained using query cases as input and extracted legal elements from those cases as output. This method is noted for being less noisy, cost-effective, and better at handling long-tail crime types.
  • Constrained Decoding with FM-index: During inference, LEGALSEARCHLM uses constrained beam decoding with an FM-index. This ensures that generated content is grounded in the corpus, effectively linking generated elements back to source documents for retrieval.

Datasets

  • LEGAR BENCH: The paper introduces LEGAR BENCH, a new large-scale Korean LCR benchmark. It comprises two versions:
  • LEGAR BENCHStandard: Covers 411 diverse crime types with queries over 1.2 million candidate cases. Crime types are systematically constructed based on statutory provisions, offering finer granularity than official charge titles.
  • LEGAR BENCHStricter: Focuses on stricter relevance criteria, considering factual details and legal issues that influence judgments. It annotates 170K cases across 160 crime types using 102 crime-specific legal factors and 443 options.
  • Existing Benchmarks: The paper mentions COLIEE2024, LeCaRD, and LeCaRDv2 for comparison, highlighting how LEGAR BENCH expands retrieval pool and query coverage.

Key Insights and Ideas

  • Legal Elements as Keys: The core idea is that generating specific legal elements (atomic facts influencing judgment) is more effective for retrieval than relying on general embeddings or simple keyword matching.
  • Generative Retrieval for LCR: Framing LCR as a generative task, specifically generating legal elements, offers a novel way to overcome the limitations of traditional retrieval methods in the legal domain.
  • First-Token-Awareness is Crucial: The paper emphasizes that the initial tokens generated by a retrieval model can significantly impact the relevance of the retrieved documents, especially in specialized domains like law.
  • Generalization through Legal Elements: LEGALSEARCHLM's ability to generalize to unseen crime types by focusing on capturing key legal elements, rather than just memorizing patterns from specific identifiers, is a significant finding.

Key Results

  • LEGAR BENCHStandard Performance: LEGALSEARCHLM outperforms baselines by up to 20% on LEGAR BENCHStandard, achieving state-of-the-art performance. It surpasses BM25 by 17%, Contriever by 20%, and SAILER by 6%.
  • LEGAR BENCHStricter Performance: On the stricter version, LEGALSEARCHLM demonstrates robustness across varying difficulty levels, outperforming embedding-based retrievers like SAILER, which degrade significantly on harder problems.
  • Generalization: LEGALSEARCHLM trained on a subset of data (sexual crimes) outperforms a naive generative model trained on all data by 15.66% on unseen crime types, highlighting the effectiveness of its legal element-focused approach.

Limitations and Future Work

  • Korean Legal System Focus: The LEGAR BENCH dataset is restricted to Korean legal cases and statutes, which may limit its applicability to other jurisdictions or non-Korean speakers.
  • Data Noise: Despite expert involvement, the dataset may contain undetected noise due to the complexity of legal relevance judgments.
  • Future Work: The authors suggest exploring further domain-specific adaptations for generative retrieval models and expanding the benchmark to include more diverse legal domains and languages.

Summary updated

BibTeX
@inproceedings{kim2025legalsearchlm,
  author = {Chaeeun Kim and Jinu Lee and Wonseok Hwang},
  title = {{LegalSearchLM: Rethinking Legal Case Retrieval as Legal Elements Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure