On the Role of Negative Precedent in Legal Outcome Prediction

Josef Valvoda, Ryan Cotterell, Simone Teufel

Abstract: Every legal case sets a precedent by developing the law in one of the following two ways. It either expands its scope, in which case it sets positive precedent, or it narrows it, in which case it sets negative precedent. Legal outcome prediction, the prediction of positive outcome, is an increasingly popular task in AI. In contrast, we turn our focus to negative outcomes here, and introduce a new task of negative outcome prediction. We discover an asymmetry in existing models' ability to predict positive and negative outcomes. Where the state-of-the-art outcome prediction model we used predicts positive outcomes at 75.06 F1, it predicts negative outcomes at only 10.09 F1, worse than a random baseline. To address this performance gap, we develop two new models inspired by the dynamics of a court process. Our first model significantly improves positive outcome prediction score to 77.15 F1 and our second model more than doubles the negative outcome prediction performance to 24.01 F1. Despite this improvement, shifting focus to negative outcomes reveals that there is still much room for improvement for outcome prediction models.

Introduction

In Legal NLP, "outcome prediction" has traditionally focused on identifying whether a court found a violation of a specific legal article. However, this focus ignores the adversarial nature of the judicial system. A court case involves a claimant alleging a breach of rights and a judge deciding the validity of that claim.

When a judge rejects a lawyer's specific claim, it creates a negative outcome. These outcomes are legally binding precedents that narrow the scope of the law. Current AI models often treat a "non-violation" as a simple binary zero, failing to distinguish between a law that was never claimed and a claim that was explicitly rejected. This paper introduces the task of negative outcome prediction and addresses the massive performance gap where standard models perform worse than random chance at identifying these rejected claims.

Methodology

To bridge the performance gap, we move beyond "black-box" classification toward models that incorporate structural priors of the judicial process. We propose two primary architectures built upon transformer backbones (BERT, LEGAL-BERT, and Longformer):

  1. The Joint Model: This model treats the claim and the outcome as a joint distribution. It utilizes a 3-way classification for each legal article: (Positive Outcome, Claimed), (Negative Outcome, Claimed), or (Null Outcome, Not Claimed).
  2. The Claim–Outcome Model: This architecture explicitly mimics trial sequencing. It first predicts the probability of a lawyer making a specific claim. Conditioned on that claim being made, it then predicts whether the judge found a violation (+) or a rejection (-).
Architectural comparison of baseline models (a, b) versus the proposed Joint Model (c) and Claim–Outcome Model (d).

Both models utilize a "Law-Abiding" framework, enforcing the logical constraint that a judge cannot provide an outcome for a law that was not claimed by the legal counsel.

Experimental Results

Our experiments on European Court of Human Rights (ECtHR) cases reveal a stark asymmetry in predictive capabilities. While standard models achieve ~75 F1 on positive outcomes, they struggle significantly with negative outcomes (~10 F1). This is likely due to the "distinguishing" problem: while positive outcomes often involve applying clear precedents, negative outcomes require identifying subtle factual nuances that lead a judge to distinguish a case from previous successful ones.

Performance comparison (F1 score) across different transformer backbones, highlighting the significant boost provided by the Claim–Outcome model (red bars) in negative outcome prediction.

The Claim–Outcome model achieved a breakthrough in negative outcome prediction, reaching 24.01 F1, more than doubling the performance of state-of-the-art baselines. Additionally, the Joint Model improved positive outcome prediction to 77.15 F1. Our results suggest that the structural relationship between claims and outcomes is a more significant factor in performance than the specific choice of language model pre-training.

Discussion and Future Work

Despite doubling the performance on negative outcomes, an F1 score of 24 indicates that legal AI still lacks a deep understanding of judicial reasoning. Future research should focus on modeling the logic of "distinguishing" cases—how models can learn to identify the specific factual differences that lead to rejections. Furthermore, relaxing the assumption of independence between legal articles could better reflect how lawyers bundle related claims in real-world litigation.

Summary updated

Abstract

Every legal case sets a precedent by developing the law in one of the following two ways. It either expands its scope, in which case it sets positive precedent, or it narrows it, in which case it sets negative precedent. Legal outcome prediction, the prediction of positive outcome, is an increasingly popular task in AI. In contrast, we turn our focus to negative outcomes here, and introduce a new task of negative outcome prediction. We discover an asymmetry in existing models' ability to predict positive and negative outcomes. Where the state-of-the-art outcome prediction model we used predicts positive outcomes at 75.06 F1, it predicts negative outcomes at only 10.09 F1, worse than a random baseline. To address this performance gap, we develop two new models inspired by the dynamics of a court process. Our first model significantly improves positive outcome prediction score to 77.15 F1 and our second model more than doubles the negative outcome prediction performance to 24.01 F1. Despite this improvement, shifting focus to negative outcomes reveals that there is still much room for improvement for outcome prediction models.
BibTeX
@article{valvoda2023role,
  author = {Josef Valvoda and Ryan Cotterell and Simone Teufel},
  title = {{On the Role of Negative Precedent in Legal Outcome Prediction}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=dCOSgazG1g},
}
Figure