Automatic Judgement Forecasting for Pending Applications of the European Court of Human Rights

Masha Medvedeva, Ahmet Üstün, Xiao Xu, Michel Vols, Martijn WielingASAIL/LegalAIIA@ICAIL 2021 · OpenReview

Introduction

Natural Language Processing (NLP) has made significant strides in legal analytics, with many studies claiming high accuracy in predicting judicial decisions. However, a critical examination reveals a methodological flaw: most models rely on the "Facts" section of final judgements. These summaries are often written post-hoc, potentially incorporating hindsight bias that simplifies the prediction task. This research distinguishes between classification (post-hoc labeling) and forecasting (predicting outcomes of pending cases) to provide a more realistic assessment of legal AI capabilities.

Methodology

The study evaluates three distinct modeling approaches:

  1. Linear SVM: A baseline using n-gram features (sequences of 1–5 words).
  2. Hierarchical BERT (H-BERT): A transformer model adapted for long documents by processing facts individually before aggregation via an attention layer.
  3. LEGAL-BERT: A BERT variant pre-trained on a large-scale legal corpus to capture domain-specific semantics.

The models were tested across three temporal setups (data up to 2017, 2018, and 2019) to ensure longitudinal robustness.

Dataset: The ECtHR Benchmark

A new benchmark dataset was curated from the European Court of Human Rights (ECtHR) HUDOC database. Crucially, it includes "communicated cases"—summaries published when the court deems an application admissible but before a final ruling is made. This allows for true forecasting by using only information available to the parties at the time of communication. The dataset comprises raw and pre-processed text for final judgements, admissibility decisions, and communicated cases.

Results and Discussion

The experiments reveal a stark contrast between classification and forecasting performance, highlighting the "optimism bias" prevalent in current legal AI research.

The Hindsight Bias and Performance Gap

When models are trained on final judgements (classification), they achieve high Macro F1-scores, often exceeding 0.90. However, when tasked with forecasting using communicated cases, performance drops significantly.

Performance metrics for the classification task (2019 Final Judgements), showing high F1 scores across all models.
Performance metrics for the forecasting task (2019 Communicated Cases), illustrating the significant drop in F1 scores compared to classification.

This discrepancy suggests that final judgements contain "spoiler" information—details emphasized by judges to justify their eventual ruling—which makes the classification task artificially easy. In contrast, forecasting involves inherent ambiguity and incomplete information, representing a much harder challenge for current NLP architectures. While classification scores often soar into the 80s and 90s, true forecasting performance struggles to break past the mid-60s.

Model Sophistication vs. Task Difficulty

Interestingly, while transformer-based models (H-BERT, LEGAL-BERT) significantly outperform the SVM in classification, they offer little to no advantage in the forecasting task. This suggests that the primary bottleneck in legal forecasting is the quality and nature of the input data rather than the model's complexity. Furthermore, all models performed significantly worse on 2020 cases, suggesting that these systems may be "brittle" when faced with evolving legal landscapes or social developments.

Conclusion

True legal forecasting is significantly more difficult than post-hoc classification. The introduction of the ECtHR benchmark provides a necessary reality check for the field, highlighting the impact of hindsight bias. Future research should focus on improving feature selection from early-stage documents and enhancing model explainability to assist legal professionals rather than attempting to replace judicial discretion.

Summary updated

BibTeX
@inproceedings{medvedeva2021automatic,
  author = {Masha Medvedeva and Ahmet Üstün and Xiao Xu and Michel Vols and Martijn Wieling},
  title = {{Automatic Judgement Forecasting for Pending Applications of the European Court of Human Rights}},
  booktitle = {ASAIL/LegalAIIA@ICAIL},
  year = {2021},
  url = {https://openreview.net/forum?id=vOV4fXPchW},
}
Figure