Introduction
In the legal profession, the accuracy of information is not merely a technical requirement but a professional necessity. As Large Language Models (LLMs) are increasingly integrated into legal workflows, there is a growing interest in using them as "knowledge bases" (LM-as-KB) for retrieving facts about case law and legislation. However, the risk of "hallucinations"—the generation of plausible-sounding but false information—poses a significant liability. This paper evaluates the factuality of several open-source LLMs in the legal domain and explores how domain-specific pre-training and prompting strategies can mitigate errors.
Methodology and Dataset
The researchers evaluated seven open-source models, all under 8 billion parameters to ensure they could be deployed on local, private hardware—a common requirement for legal data privacy. The models included general-purpose LLMs like Llama-3, Mistral-7B, and Gemma, as well as SaulLM, a model specifically further pre-trained on legal corpora.
To test these models, the authors developed a new benchmark dataset comprising 8,920 question-answer pairs. These pairs cover critical legal relations, such as "applies to jurisdiction," "majority opinion by," and "legislated by." Unlike earlier benchmarks that relied on simple, one-token facts, this dataset leverages the Wikipedia Category Graph (WCG) and Wikidata to extract structured, atomic facts about complex legal entities, providing a more rigorous and realistic test of legal knowledge.
Evaluation Strategies
A primary contribution of this research is the identification of "Metric Sensitivity." Traditional benchmarks often use Exact Matching (EM), which requires the model's output to match the ground-truth label perfectly. The authors argue that this penalizes LLMs for being helpful or verbose (e.g., answering "Justice Alito" instead of the label "Samuel Alito"). To address this, they compared three strategies:
- Exact Matching (EM): Strict string equality.
- Alias Matching (AM): Matches against the label or known synonyms (e.g., "USA" vs. "United States").
- Fuzzy Matching (FM): Considers the answer correct if it contains the label or an alias.
Additionally, the study tested the impact of few-shot prompting (providing five examples) and an "abstain" instruction, which explicitly permits the model to state "I don't know" when uncertain.
Key Insights
1. The Impact of Verbosity
Under strict Exact Matching, many models appeared to have near-zero accuracy. However, performance jumped significantly under Fuzzy Matching. For SaulLM in few-shot mode, precision rose from 36% (EM) to 81% (FM). This suggests that LLMs often possess the correct underlying knowledge but fail to conform to the rigid formatting expected by standard evaluation metrics.
2. The Power of Abstention
One of the most effective ways to increase precision in high-stakes environments is to allow the model to remain silent. When given an "abstain" instruction, models like SaulLM chose not to answer approximately 40% of the time in few-shot scenarios. This selective silence directly contributed to a significant boost in the precision of the remaining answers.
3. Overcoming Priors with Few-Shot Learning
Without examples, models often default to common web patterns that may be factually incorrect in a legal context. For instance, models might reverse "Plaintiff v. Defendant" based on how cases are colloquially discussed online. Providing just five examples in the prompt helps the model override these incorrect priors and adopt the expected output format.
4. The Necessity of Domain Pre-training
Domain-specific training proved essential. SaulLM outperformed its base model (Mistral-7B) by a wide margin, increasing precision from 63% to 81%. This demonstrates that while general-purpose models are powerful, they lack the specialized depth required for professional legal applications.
Conclusion and Future Work
While achieving 81.2% precision represents a significant step forward, the authors emphasize that this level of accuracy is still insufficient for high-stakes legal tasks where the goal is 100% reliability. The current study focused on atomic facts; future research is required to evaluate how these models handle complex legal reasoning and the application of rules to specific disputes. Furthermore, more sophisticated post-processing is needed to ensure that Fuzzy Matching does not inadvertently credit a model for a correct term buried within a fundamentally flawed explanation.
Summary updated