Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models

Matthew Dahl, Varun Magesh, Mirac Suzgun, Daniel E. Ho

Introduction

Large Language Models (LLMs) are increasingly marketed as transformative tools for the legal industry, often described as the future of "AI lawyers." However, a systematic empirical study reveals a sobering reality: these models frequently hallucinate legal facts at alarming rates. Even state-of-the-art models like GPT-4 fail to provide accurate information more than half the time on complex legal queries. This performance gap is not uniform; it widens significantly as tasks move from high-profile Supreme Court cases to the more localized district court litigation that constitutes the bulk of legal practice.

The legal profession is built on the principle of stare decisis—the requirement to follow precedent. In this domain, factual accuracy is a fundamental requirement rather than a preference. While high-profile instances of AI-generated fake cases have made headlines, this research provides a systematic, empirical understanding of how often and why LLMs fail in the legal domain. The study moves beyond simple "closed-domain" tasks, such as summarization, to "open-domain" queries that test the models' internal knowledge of American case law.

Methodology

The researchers developed a framework of 14 legal knowledge tasks categorized by complexity:

  • Low Complexity: Basic metadata, such as whether a case exists or who authored the opinion.
  • Moderate Complexity: Substantive facts, such as whether a court affirmed or reversed a lower ruling, or providing verbatim quotations.
  • High Complexity: Legal reasoning tasks, including determining a case's central holding or assessing doctrinal agreement between two cases.

Evaluation utilized two distinct strategies. For tasks with a clear "ground truth," the researchers used reference-based querying against curated legal databases. For more complex reasoning tasks where a single correct string does not exist, they employed a contradiction detection approach. By querying a model multiple times at a higher "temperature" (stochasticity), they identified hallucinations when the model provided self-contradictory answers.

The study benchmarked four major models—OpenAI’s GPT-4 and GPT-3.5, Google’s PaLM 2, and Meta’s Llama 2—using a massive sample of 15,000 cases drawn from the Caselaw Access Project (CAP), the Supreme Court Database (SCDB), and Shepard’s Citations.

Key Results

The findings indicate that legal hallucinations are pervasive across all tested models. When asked direct questions about federal cases, the mean hallucination rates were consistently high, posing significant risks for legal practitioners relying on these tools.

As shown in the data, GPT-4 demonstrated the lowest hallucination rate at 58%, while Llama 2 reached a staggering 88%. Beyond simple facts, models struggled significantly with "Doctrinal Agreement" tasks, performing near the level of random guessing. Furthermore, geographic variance was observed; models performed significantly better on cases from the 2nd and 9th Circuits (New York and California), likely due to the higher volume of legal data originating from these hubs.

Insights into Model Behavior

The Hierarchy of Hallucination

Models exhibit a "trickle-down" knowledge gap. They are most accurate regarding Supreme Court (SCOTUS) precedents but fail significantly on District Court (USDC) cases. This suggests that LLMs internalize famous national precedents while remaining largely ignorant of the localized law that governs the vast majority of actual litigation.

The "Monoculture" Risk and Sycophancy

LLMs show a strong bias toward newer, highly cited cases and well-known judges. This creates a risk of a legal "monoculture" where AI tools ignore older or less-cited—but still binding—precedents. Additionally, models exhibit "sycophancy," or contra-factual bias, where they uncritically accept a user's incorrect legal assumptions. If prompted with a false premise, models often "hallucinate" a response to satisfy the user's query rather than correcting the error.

The Calibration Paradox

While GPT-4 is the most accurate model, it is often less "calibrated" than models like PaLM 2. Calibration refers to the correlation between a model's confidence and its actual accuracy. GPT-4 frequently hallucinates with high conviction, making it potentially more misleading than a model that "knows what it doesn't know."

Conclusion and Future Directions

This study establishes a lower bound for legal hallucinations, as the contradiction detection method only catches inconsistent errors. While Retrieval-Augmented Generation (RAG) is often proposed as a solution, the authors caution that its effectiveness depends on the retrieval system's ability to distinguish between binding and overruled precedents. To support further research, the authors have released their evaluation data on HuggingFace (reglab/legal_hallucinations) to serve as a benchmark for the development of more reliable legal AI.

Summary updated

Abstract

Abstract Do large language models (LLMs) know the law? LLMs are increasingly being used to augment legal practice, education, and research, yet their revolutionary potential is threatened by the presence of “hallucinations”—textual output that is not consistent with legal facts. We present the first systematic evidence of these hallucinations in public-facing LLMs, documenting trends across jurisdictions, courts, time periods, and cases. Using OpenAI’s ChatGPT 4 and other public models, we show that LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users’ incorrect legal assumptions. We conclude by cautioning against the rapid and unsupervised integration of popular LLMs into legal tasks, and we develop a typology of legal hallucinations to guide future research in this area.
BibTeX
@article{dahl2024large,
  author = {Matthew Dahl and Varun Magesh and Mirac Suzgun and Daniel E. Ho},
  title = {{Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models}},
  journal = {The Journal of Legal Analysis},
  year = {2024},
  doi = {10.1093/jla/laae003},
}
Figure