Rethinking the Development of Large Language Models from the Causal Perspective: A Legal Text Prediction Case Study

Haotian Chen, Lingwei Zhang, Yiran Liu, Yang YuAAAI 2024 · DOI

Introduction

Large Language Models (LLMs) have demonstrated remarkable success in specialized domains like legal judgment prediction. However, this success often masks a dangerous underlying reality: these models frequently succeed by "gaming" spurious correlations rather than understanding legal causality. In high-stakes fields, relying on non-semantic shortcuts—such as punctuation patterns or specific writing styles—can lead to catastrophic failures in real-world applications.

In legal text prediction, a model might correctly predict a court's decision for entirely the wrong reasons. Research into state-of-the-art models like Legal-BERT reveals a startling reliance on function words (e.g., "the", "of") and punctuation marks.

Analysis of Legal-BERT's reliance on non-semantic tokens. The model maintains over 92% performance using only 5% of words (primarily punctuation and function words) and exhibits extreme sensitivity to irrelevant character masks.

As shown in the analysis above, Legal-BERT preserves 92.13% of its original performance even when restricted to the top 5% of its "evidence words." These words are predominantly non-legal tokens. Furthermore, the model's fragility is exposed by simple adversarial changes; for instance, masking a single bracket [ in a fact description can completely reverse a predicted violation. This behavior stems from data imbalance and the i.i.d. (independent and identically distributed) assumption, where models greedily absorb any correlation that happens to hold true in the training set.

A Three-Pronged Causal Framework

To move beyond simple correlation, this study proposes a framework designed to "infuse causality" into the model development pipeline. This approach addresses three core problems: data bias, architectural limitations, and flawed model selection.

Conceptual framework for infusing causality into the legal prediction pipeline, addressing data bias, architectural limitations, and evaluation metrics to isolate true rationales.

The proposed methodology consists of three key components:

  1. Graph Construction via OIE: Using Open Information Extraction (OIE) and coreference resolution, the system extracts "causal triplets" (subject-predicate-object) to filter out non-causal noise and focus on the core facts of a case.
  2. Causality-Aware Self-Attention Mechanism (CASAM): Standard self-attention is modified using an adjacency matrix derived from the OIE graph. This mechanism reallocates attention weights to ensure the model prioritizes interactions between words with documented causal links.
  3. Causality-Based Model Selection: Rather than relying solely on standard accuracy, the framework evaluates models using eight types of legal-specific adversarial attacks. This measures the "Certified Ratio" (CR)—the percentage of predictions that remain consistent under perturbations that should not legally affect the outcome.

Key Results and Insights

The implementation of CASAM yielded significant improvements across prominent benchmarks, including ECtHR (European Court of Human Rights) and LEDGAR (contract provision classification).

  • Performance Gains: CASAM achieved new state-of-the-art results, improving the Macro-F1 score on ECtHR Task A from 66.0 to 68.5.
  • Adversarial Robustness: The most dramatic improvements were seen in robustness. While standard models often fail when functional words are masked, CASAM maintained a certified ratio of over 99% in many attack scenarios, significantly outperforming standard BERT and RoBERTa architectures.
  • Causal Intervention: Ablation studies confirmed that the attention intervention is the primary driver of success; removing the causal component caused performance to revert to baseline levels.

Conclusion

This research demonstrates that high F1 scores on standard benchmarks can be deceptive in the legal domain. True model reliability requires moving from correlation-based learning to causal intervention. While currently limited by the precision of OIE tools, the CASAM framework provides a robust path forward for developing AI systems that actually "read" the law rather than finding statistical shortcuts. Future work will explore scaling these causal perspectives to larger models and other high-stakes domains.

Summary updated

BibTeX
@inproceedings{chen2024rethinking,
  author = {Haotian Chen and Lingwei Zhang and Yiran Liu and Yang Yu},
  title = {{Rethinking the Development of Large Language Models from the Causal Perspective: A Legal Text Prediction Case Study}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i19.30086},
}
Figure