Detecting Legal Citations in United Kingdom Court Judgments

Holli Sargeant, Andreas Östling, Måns MagnussonEMNLP 2025

TL;DR

This paper tackles the challenging task of automatically identifying legal citations (legislation and case law) in UK court judgments. They introduce a new, high-quality annotated dataset and systematically compare rule-based methods, transformer encoders, and large language models, finding that transformer models, particularly ModernBERT, achieve state-of-the-art performance.

What Problem Does This Paper Address?

Legal citation detection is crucial for many downstream NLP tasks in law, such as mapping legal precedents, analyzing citation networks, and improving document retrieval. However, accurately extracting these citations from UK court judgments is particularly difficult due to:

  • Evolving citation styles: UK citation practices have changed significantly over centuries.
  • Varied formats: Citations can appear in natural prose, with abbreviations, and include references to foreign or historical authorities.
  • Complexity of legal language: Specialized terminology and intricate cross-referencing conventions add to the challenge.

While previous work has addressed legal entity recognition (LER) in various jurisdictions, there's a notable gap in systematic studies specifically for UK law. This paper aims to fill that gap by providing a comprehensive evaluation of different LER approaches on UK legal texts.

Methods

The authors compare three main paradigms for legal citation detection:

  1. Rule-Based Regular Expressions (RegEx): They developed custom RegEx patterns, leveraging legal expertise to capture common citation formats for both legislation and case law. These patterns were iteratively refined.
  2. Pre-trained Transformer Encoder Models: They fine-tuned several encoder models, including BERT, RoBERTa, LEGAL-BERT (specifically trained on legal corpora), and ModernBERT (a more recent architecture with enhanced context length and efficiency). They trained separate models for UK-only references and for all legal references.
  3. Large Language Models (LLMs): They evaluated GPT-4.1 using a few-shot prompting strategy, aiming for token-level reference extraction. They also explored a retrieval-augmented generation (RAG) approach to tailor prompts to specific contexts.

Performance was evaluated using F1-score, precision, recall, Jaccard Index, and Seqeval F1.

Datasets

The primary dataset used is the Cambridge Law Corpus (CLC), specifically a subset annotated for this study, named CLC-Citation.

  • CLC-Citation: This is a newly curated, expert-annotated dataset comprising 190 UK court judgments. It contains 45,179 fine-grained annotations for UK and non-UK legislation and case references. The authors developed a detailed annotation schema to differentiate UK and non-UK entities and ensure consistency. This dataset is released to the research community.

Key Insights and Ideas

  • Transformer models dominate: Transformer-based encoder models significantly outperform both RegEx and LLMs on this task, highlighting their suitability for complex legal text.
  • ModernBERT shines: The newer ModernBERT architecture achieved the highest F1 score, suggesting that advancements in transformer design are beneficial for legal citation detection.
  • RegEx limitations are stark: The performance of RegEx baselines is very low, underscoring the inadequacy of rule-based approaches for the complexity and variability of UK legal citations.
  • LLMs show promise but have drawbacks: GPT-4.1 demonstrated high recall but struggled with precision, leading to a lower overall F1 score and higher costs, making it less practical for large-scale applications currently.
  • Domain-specific pre-training is helpful, but not a silver bullet: LEGAL-BERT, pre-trained on legal data, performed well but was slightly outperformed by ModernBERT, which might be due to LEGAL-BERT's pre-training corpus lacking sufficient UK case law.
  • UK-specific challenges persist: Historical legal references and academic citations (which can resemble legal ones) pose particular challenges for all models.

Key Results

  • ModernBERT achieved the highest macro-averaged F1 score of 93.3% on the full label set.
  • Other transformer encoder models (BERT, RoBERTa, LEGAL-BERT) also achieved F1 scores exceeding 90%.
  • The strongest RegEx baseline achieved an F1 score of 35.42%.
  • GPT-4.1 achieved an F1 score of 76.57% (dynamic model), with high recall (96.74%) but significantly lower precision (63.36%).
  • Transformer models generally performed better on case law references than on statute references, primarily due to higher recall.

Limitations and Future Work

  • Dataset Scope: The CLC-Citation dataset, while comprehensive, comprises only 190 UK court judgments. A larger and more diverse dataset covering more courts, time periods, and case types would be beneficial.
  • Compute Cost: Experiments with commercial LLMs like GPT-4.1 are prohibitively expensive for large-scale applications. Direct access to embeddings for token prediction would be more cost-effective.
  • Historical and Non-UK Citations: All models, especially transformers, struggle with historical legal references and citations from other jurisdictions that deviate from modern norms.
  • Tokenization: Standard tokenizers may not be optimal for legal text.
  • Future Work:
    • Developing more tailored domain adaptation for UK law, potentially creating a "CLC-ModernBERT."
    • Addressing the limitations of standard tokenizers.
  • Exploring interdisciplinary collaboration to improve citation detection models.
  • Further research into handling historical and non-UK citations.

Summary updated

BibTeX
@inproceedings{sargeant2025detecting,
  author = {Holli Sargeant and Andreas Östling and Måns Magnusson},
  title = {{Detecting Legal Citations in United Kingdom Court Judgments}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure