How well do SOTA legal reasoning models support abductive reasoning?

Ha-Thanh Nguyen, Randy Goebel, Francesca Toni, Kostas Stathis, Ken Satoh

Abstract

While Large Language Models (LLMs) have demonstrated significant proficiency in legal text retrieval and summarization, their capacity for the complex logical reasoning required in legal practice remains under-examined. This paper investigates the performance of state-of-the-art (SOTA) models on abductive reasoning—the process of forming explanatory hypotheses from observations. We introduce L’ART (Logic-augmented Abductive Reasoning Dataset), a dataset of nearly 500,000 samples designed to test logical validity rather than simple plausibility. Our findings reveal a "Legal Bias Paradox": models specifically pre-trained on legal corpora often underperform compared to general-purpose models, with many large-scale models performing no better than random chance.

Introduction

In the legal domain, intelligence is defined by more than the ability to locate relevant precedents; it requires the construction of coherent legal theories. This necessitates abductive reasoning: the inference of a hypothesis ($H$) that, when combined with an initial observation ($O_1$), logically leads to a second observation ($O_2$). Despite the hype surrounding "AI lawyers," current benchmarks often fail to distinguish between true logical reasoning and high-dimensional pattern matching.

Existing datasets for abductive reasoning, such as the original ART dataset, frequently rely on crowdsourced "common sense" narratives. These are often logically inconsistent or contain linguistic biases that allow models to achieve high scores without performing actual reasoning. To address this, we propose a more rigorous evaluation framework and a massive, logic-augmented dataset to benchmark the current state of legal AI.

Methodology

Task Redefinition: From Plausibility to Validity

Standard abductive Natural Language Inference ($\alpha NLI$) tasks typically present a model with two observations and ask it to choose the more plausible of two hypotheses. We argue this setup is prone to overfitting. We redefine the task as $\alpha NLI^*$, a binary validation task. Given a triple $(O_1, H, O_2)$, the model must determine if the relationship is logically valid, specifically satisfying the condition $O_1 \land H \implies O_2$.

Logic-Augmented Generation

To ensure the dataset's integrity, we employed a two-pronged generation strategy:

  1. Theorem Proving: We utilized a logic-based theorem generator to create synthetic, logically sound samples that are free from the "enthymemes" (unstated assumptions) common in human speech.
  2. Observation-Hypothesis Interchangeability: Recognizing that $O_1$ and $H$ are functionally interchangeable in a logical conjunction, we used this property to double the volume of valid samples while maintaining consistency.
  3. Rigorous Negation: To prevent models from relying on simple keyword cues, we used truth tables to identify specific components for negation, creating challenging negative samples that require precise logical parsing to identify as invalid.

The L’ART Dataset

The resulting Logic-augmented Abductive Reasoning Dataset (L’ART) contains 498,697 samples. This represents a 2.5x increase in scale over the original ART dataset, providing a more robust foundation for training and evaluating deep learning architectures on formal logic.

Experimental Results

We evaluated several SOTA models, including general-purpose transformers and those specifically pre-trained on legal data. The results, summarized below, indicate a significant gap between current AI capabilities and the requirements of legal reasoning.

Model Test Accuracy
BERT Base 61.62%
BERT-PLI 61.15%
Legal BERT 56.19%
BERTLaw 53.71%
BERT Large 50.00%
GPT-3 (Zero-shot) 49.59%

A surprising finding was that models pre-trained specifically on legal text (Legal BERT, BERTLaw) performed worse than the general-purpose BERT Base. This suggests that legal pre-training, which focuses heavily on retrieval and linguistic style, may introduce biases that degrade a model's ability to perform pure logical abduction. The models appear to prioritize legalistic vocabulary over logical structure.

The Failure of Scale

Contrary to the "scaling laws" observed in other NLP tasks, increasing model size did not improve performance. BERT Large and GPT-3 (in a zero-shot configuration) achieved accuracies near 50%, essentially equivalent to a coin flip. This suggests that the ability to perform abductive logic is not an emergent property of model scale or next-token prediction density.

Discussion: The Enthymeme Problem

Human reasoning is often shorthand, relying on unstated background knowledge (enthymemes). By using a theorem generator to strip away these hidden assumptions, L’ART forces models to rely entirely on explicit logic. The failure of SOTA models on this dataset highlights their reliance on linguistic shortcuts rather than a functional understanding of logical entailment.

Conclusion

Current legal AI models are proficient at processing text but struggle significantly with the abductive reasoning central to legal argument. To bridge this gap, future research should focus on:

  • New Pre-training Objectives: Moving beyond next-token prediction to include objectives that target logical consistency.
  • Neuro-symbolic Architectures: Integrating symbolic logic with the linguistic flexibility of deep learning.
  • Balanced Training: Investigating how to train legal models without sacrificing general logical reasoning capabilities.

While AI may assist in legal research, the "AI lawyer" capable of constructing complex legal theories remains a distant prospect.

Summary updated

BibTeX
@inproceedings{nguyen2023how,
  author = {Ha-Thanh Nguyen and Randy Goebel and Francesca Toni and Kostas Stathis and Ken Satoh},
  title = {{How well do SOTA legal reasoning models support abductive reasoning?}},
  booktitle = {ICLP Workshops},
  year = {2023},
}
Figure