Predicting Legal Outcomes from Pre-decisional Facts

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 216 papers31 conceptsfrozen 2026-10-04

In the field of Legal Judgment Prediction (LJP), researchers have long pursued the "AI Judge"—a model capable of reading a case and predicting its verdict. On paper, these models appear remarkably successful, often achieving high accuracy. However, a growing body of research suggests that this success is frequently an illusion caused by retrospective bias.

The problem lies in the data: most legal NLP models are trained on judicial opinions. As Robert Mahari and colleagues point out in The Law and NLP: Bridging Disciplinary Disconnects, a judicial opinion is not a neutral record of facts. It is a post-hoc justification written by a judge to rationalize a decision they have already made. When a model "predicts" a verdict from such a summary, it isn't performing legal reasoning; it is simply detecting the linguistic cues of a finished conclusion.

The Trap of Post-hoc Justification

To understand why this matters, consider the "engine room" of a court case. Before a decision is reached, the facts are messy, contested, and unorganized. Once a judge decides, they curate these facts into a coherent narrative that supports the ruling. This creates a circular dependency: the input to the model (the summary) already contains the output (the verdict).

Josef Valvoda and Ryan Cotterell argue in The Ethics of Automating Legal Actors that this justification bias makes models unreliable for real-world use. If a model learns to recognize a judge's reasoning style rather than the merits of the case, it cannot help a lawyer evaluate a new, undecided case. This "alignment gap" is further explored in their 2024 study, which found that even when models predict outcomes accurately, their internal "reasoning" bears little resemblance to how human judges actually apply precedent.

Shifting to Pre-decisional Data

To break this circularity, the research direction has shifted toward pre-decisional data sourcing. This involves training models on documents that exist before the judge speaks, such as:

  • Raw Plaintiff Complaints: Gil Semo and others introduced ClassActionPrediction, a dataset using original US class action complaints. They found that while models struggle more with these "raw" inputs, they are better calibrated and more useful for actual legal triage.
  • Court Transcripts: In MSJudge, Luyao Ma and colleagues developed a framework that operates on raw transcripts and claims, treating the recognition of facts as an auxiliary task rather than a given input.
  • Primary Source Documents: Wayan Oger Vihikan and the team behind the WIPO domain dispute study emphasize that for AI to assist in legal triage, it must perform on documents that do not yet contain the arbitrator's synthesized perspective.

Blinded Annotation and Selective Prediction

The challenge of objectivity extends to human experts as well. Isabella Risini and Matthias Grabmair found that human annotators identify different "relevant" facts depending on whether they are "blinded" to the final verdict. This suggests that even the gold-standard data used to train models can be tainted by the knowledge of the outcome.

To make these systems safer, researchers like T. Y. S. S. Santosh are moving toward selective prediction. In The Craft of Selective Prediction, they argue that a model should "abstain" from a decision when it is uncertain. By masking artifacts and focusing on substantive legal analysis rather than spurious correlations in post-hoc summaries, these models become more trustworthy partners for human legal experts.

The goal is no longer just to predict the winner of a case, but to build systems that understand the facts as they were first presented—before the finality of a judgment smoothed over the complexities of the law.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 9 papers
  2. Version 1 · 2026-10-04 · 16 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_post_hoc_justification_bias_9454baf5,
  title = {post-hoc justification bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/post-hoc-justification-bias/rdr_9454baf506bf4cbcb41d8973602cd7e5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_predicting_legal_outcomes_from_pre_decis_9454baf5_v1,
  title = {Predicting Legal Outcomes from Pre-decisional Facts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predicting-legal-outcomes-from-pre-decisional-facts/rdr_9454baf506bf4cbcb41d8973602cd7e5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}