LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development

Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Martin Katz, Anders Søgaard

Abstract

Legal Natural Language Processing (NLP) has historically suffered from a "jurisdictional blind spot," with most models trained predominantly on United States case law. This paper introduces LeXFiles, a 19-billion-token English legal corpus spanning six major jurisdictions, and LegalLAMA, a specialized benchmark designed to probe the legal knowledge of language models. By pre-training models on this diverse multinational dataset, the authors demonstrate that jurisdictional diversity is crucial for developing robust legal AI. Their flagship model, LexLM, outperforms existing legal and general-purpose models across upstream, probing, and downstream tasks.

1. Introduction

The legal domain presents a unique challenge for artificial intelligence. Law is not a monolithic entity; it is a collection of distinct systems—such as those of the EU, US, UK, and Canada—each characterized by its own specific vocabulary, logic, and precedents. Despite this diversity, most existing legal-oriented Pre-trained Language Models (PLMs) are "US-centric." For instance, the widely used Pile of Law dataset is heavily biased toward American legal documents. This creates a significant performance gap when these models are applied to international or non-US legal contexts.

In "LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development," Chalkidis et al. address this gap by providing the resources necessary to build and evaluate multinational legal language models. Their work focuses on whether models truly "know" the law or are simply proficient at statistical word prediction.

2. The LeXFiles Corpus and LegalLAMA Benchmark

To facilitate the development of more globally aware models, the researchers introduced two primary resources:

2.1 LeXFiles

LeXFiles is a massive, diverse English legal corpus containing approximately 19 billion tokens across 6 million documents. It is organized into 11 sub-corpora covering legislation and case law from six diverse legal systems:

  • The European Union (EU)
  • The Council of Europe (CoE)
  • Canada
  • The United States (US)
  • The United Kingdom (UK)
  • India

2.2 LegalLAMA

To move beyond simple performance metrics, the authors developed LegalLAMA, a probing benchmark consisting of eight sub-tasks. Based on the "LAnguage Models Analysis" (LAMA) framework, it requires models to fill in the blanks for specific legal concepts (e.g., identifying ECHR article numbers or specific US contract clauses). Notably, the authors extended this framework to handle multi-token legal terms, such as "drug trafficking," allowing for a more sophisticated assessment of a model's conceptual understanding.

3. Methodology

The researchers employed a "warm-start" approach for model training. Rather than training from scratch, they utilized existing RoBERTa (Base and Large) architectures and performed an additional 1 million pre-training steps on the LeXFiles corpus.

Key technical refinements included:

  • Legal-Specific Tokenizer: A new 50k Byte-Pair Encoding (BPE) tokenizer was trained specifically on legal text to better capture domain-specific jargon.
  • Exponential Smoothing: To prevent the model from being overwhelmed by the massive volume of US case law, the authors used sampling smoothing to ensure smaller jurisdictions (like Canadian legislation) were adequately represented during training.

4. Key Insights and Results

The study yielded several critical insights into the nature of legal NLP:

4.1 The Specialization Trap

The authors found that models trained exclusively on US data (e.g., PoL-BERT) often performed worse than generic models when applied to non-US tasks. This suggests that over-specialization in a single jurisdiction can actively harm a model's multinational utility.

4.2 Knowledge Acquisition

There is a strong correlation between a model's performance in Masked Language Modeling (MLM)—predicting hidden words—and its performance on LegalLAMA probing tasks. This indicates that the "upstream" pre-training phase is where models actually acquire their foundational legal knowledge base.

4.3 Performance of LexLM

The authors' flagship model, LexLM-L (Large), demonstrated superior performance across all metrics:

  • Upstream Accuracy: Achieved 81.8% average accuracy across all LeXFiles sub-corpora, a 3% lead over the nearest competitor.
  • Probing: LexLM-L dominated nearly every category in LegalLAMA, showing a deeper grasp of multinational legal concepts.
  • Downstream Tasks: On the LexGLUE benchmark (covering classification and reasoning), LexLM-L achieved a Micro-F1 score of 73.3%, significantly outperforming both generic RoBERTa-L (71.5%) and the US-centric PoL-BERT (69.5%).

5. Limitations and Future Directions

While LeXFiles and LegalLAMA represent a major step forward, the authors note several areas for future research. The current study is limited to English-speaking jurisdictions, leaving the linguistic diversity of civil law systems in other languages unexplored. Additionally, the research focused on encoder-only models (BERT-style); as generative Large Language Models (LLMs) like GPT-4 become more prevalent, understanding how they handle multinational legal reasoning will be essential.

6. Conclusion

By introducing LeXFiles and LegalLAMA, this research provides the community with the tools to move past US-centric legal AI. The results demonstrate that while model size is important, the diversity of pre-training data is the "secret sauce" for creating robust, multinational legal language models capable of navigating the complexities of global jurisprudence.

Summary updated

BibTeX
@inproceedings{chalkidis2023lexfiles,
  author = {Ilias Chalkidis and Nicolas Garneau and Catalina Goanta and Daniel Martin Katz and Anders Søgaard},
  title = {{LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.865},
}
Figure