BLT: Can Large Language Models Handle Basic Legal Text?

Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van DurmearXiv 2023 · OpenReview

TL;DR

This paper introduces BLT, a new benchmark for evaluating Large Language Models (LLMs) on fundamental legal text comprehension tasks. They found that even state-of-the-art LLMs like GPT-4 perform surprisingly poorly on these basic tasks, highlighting a critical gap in their real-world legal applicability. However, fine-tuning a smaller LLM on BLT data achieves near-perfect performance, suggesting a path towards more reliable legal AI.

What Problem Does This Paper Address?

The legal profession is heavily text-based, and there's growing excitement about LLMs revolutionizing legal practice. While LLMs have shown promise in complex tasks like passing the bar exam, their ability to handle the most basic legal text operations – tasks that paralegals and junior lawyers perform daily – has been largely unexamined. These fundamental tasks include accurately retrieving specific text snippets from documents like depositions or contracts based on citations, or finding the citation for a given piece of text. The authors argue that without proficiency in these foundational skills, LLMs cannot be reliably deployed in legal settings, as errors in these basic operations could have significant consequences.

Methods

The core of this paper is the introduction of the Basic Legal Text (BLT) benchmark. BLT is designed to mimic simple, zero-shot legal text handling tasks. It comprises three types of legal text: deposition transcripts, synthetic hierarchical sections, and the U.S. Code. For each text type, four primary tasks are defined:

  • text→cite: Given a piece of text, find its location (page and line number, or citation).
  • cite→text: Given a location (citation), retrieve the exact text.
  • defined→cite: Given a defined term, find its definition's citation.
  • cite→defined: Given a citation, identify the term defined there.

Additionally, a fifth task, cite→amended, was introduced for the U.S. Code to test the LLM's ability to recognize and apply recently amended statutes. The benchmark is scalable, with versions ranging from BLT-4k (fitting within a 4,000 token window) to BLT-128k (fitting within a 128,000 token window), allowing for the evaluation of LLMs with varying context window sizes.

Datasets

The paper introduces the Basic Legal Text (BLT) benchmark, which is the primary dataset used for evaluation. BLT is constructed from:

  • Deposition Transcripts: A novel corpus of 33,176 lines of actual deposition transcripts obtained from federal district courts via PACER.
  • Synthetic Hierarchical Sections: Generated sections mimicking legal documents like statutes and contracts, created by repeatedly applying logical forms and using pronounceable nonces (nonsense words) to ensure novelty.
  • U.S. Code: The official compilation of U.S. federal statutes.

The authors also mention using the SARA (StAtutory Reasoning Assessment) dataset for evaluating the fine-tuned model's performance on a more complex legal task.

Key Insights and Ideas

  • LLMs struggle with basic legal text: The most striking insight is that even leading LLMs like GPT-4 and Claude perform poorly on fundamental legal text retrieval and citation tasks, despite their impressive capabilities in other domains. This suggests a significant gap between general language understanding and specialized legal text processing.
  • Fine-tuning is highly effective: Fine-tuning even a smaller model (GPT-3.5-turbo) on the BLT training data dramatically improves performance, achieving near-perfect accuracy on the benchmark tasks. This highlights the potential of targeted training for specialized domains.
  • Context window size matters, but not always predictably: Accuracy generally decreases as the prompt size (context window) increases across different BLT versions. This suggests that while LLMs are improving their ability to handle longer contexts, performance on these specific tasks can degrade.
  • Prompt phrasing can significantly impact performance: For certain tasks, like transcript text→cite, moving the question from the end to the beginning of the prompt led to substantial accuracy improvements for GPT-4. However, this effect was not consistent across all tasks, indicating task-specific sensitivity.
  • "Test-set leakage" is a concern for recent models: The authors note that newer models like GPT-4o and Claude-3.5 performed exceptionally well, potentially due to having seen the BLT test data during their training, which occurred after the BLT datasets were released online.

Key Results

  • Zero-shot performance is poor: For BLT-4k, GPT-4 achieved accuracies ranging from 78% to 98% across tasks, while GPT-4o and Claude-3.5 showed higher scores (up to 97% for Claude-3.5 on BLT-4k). However, even the best models struggled with specific tasks, like transcript cite→text, where GPT-4o achieved only 84%.
  • Fine-tuning achieves near-human performance: Fine-tuning GPT-3.5-turbo on BLT-4k training data resulted in accuracies of 98-100% across all 11 tasks, demonstrating its effectiveness.
  • Fine-tuned model excels on complex tasks: The fine-tuned GPT-3.5-turbo also showed improved performance on the SARA dataset, increasing accuracy from 54.3% to 60.9%.
  • Error analysis reveals patterns: For transcript tasks, errors often involved retrieving the line immediately before or after the correct one. For U.S. Code cite→amended, a concerning error was LLMs returning the unamended text, suggesting reliance on outdated training data.

Limitations and Future Work

  • Test-set leakage: The authors acknowledge that recent models' high performance might be inflated due to potential exposure to the BLT test data during their training.
  • "Sloppy paralegals": The paper frames current LLMs as "sloppy paralegals," implying that while they can perform some tasks, their unreliability in basic operations is a significant drawback.
  • Generalizability of fine-tuning: While fine-tuning on BLT significantly improves performance on BLT tasks and even SARA, the extent to which this fine-tuning generalizes to other complex legal tasks remains an area for further investigation.
  • Prompt engineering sensitivity: The paper highlights that prompt engineering can influence results, but they focused on basic prompt structures, suggesting more advanced prompt engineering could potentially yield further improvements.
  • Future work: The authors suggest BLT will be useful for fine-tuning LLMs for more complex legal tasks and for tracking LLM reliability in basic legal text handling. They also note that the BLT dataset is scalable and can generate unlimited new examples, allowing for continued evaluation as LLMs evolve.

Summary updated

BibTeX
@misc{blairstanek2023blt,
  author = {Andrew Blair-Stanek and Nils Holzenberger and Benjamin Van Durme},
  title = {{BLT: Can Large Language Models Handle Basic Legal Text?}},
  year = {2023},
  url = {https://openreview.net/forum?id=rPF2ijVi6D},
}
Figure