Neural Legal Judgment Prediction in English

Ilias Chalkidis, Ion Androutsopoulos, Nikolaos Aletras

Abstract

Legal judgment prediction is the task of automatically predicting the outcome of a court case, given a text describing the case's facts. Previous work on using neural models for this task has focused on Chinese; only feature-based models (e.g., using bags of words and topics) have been considered in English. We release a new English legal judgment prediction dataset, containing cases from the European Court of Human Rights. We evaluate a broad variety of neural models on the new dataset, establishing strong baselines that surpass previous feature-based models in three tasks: (1) binary violation classification; (2) multi-label classification; (3) case importance prediction. We also explore if models are biased towards demographic information via data anonymization. As a side-product, we propose a hierarchical version of BERT, which bypasses BERT's length limitation.

Introduction

Legal Judgment Prediction (LJP) is often considered the "holy grail" of legal technology. The objective is to predict a court's decision based solely on the factual description of a case. While neural network approaches have flourished in other languages, English-language research has historically relied on smaller datasets and traditional feature-based models like Support Vector Machines (SVM) with bag-of-words representations.

Two primary obstacles have hindered progress in English LJP: the lack of a large-scale, publicly available raw text dataset for training neural models, and the extreme length of legal documents. Standard transformer models like BERT are limited to 512 tokens, whereas a typical court case description can span thousands of words.

Methodology and Dataset

This study introduces the ECHR Dataset, comprising approximately 11,500 cases from the European Court of Human Rights. Each entry includes a factual summary, the specific Convention articles violated (if any), and a case importance score. To ensure realistic evaluation, the data was split temporally, using cases from 1959–2013 for training and 2014–2018 for testing.

To address the document length challenge, the researchers proposed HIER-BERT, a hierarchical adaptation of BERT. Instead of truncating the text, HIER-BERT uses BERT to encode individual facts into embeddings and then applies a self-attention mechanism over these embeddings to represent the entire case. This architecture was compared against several baselines, including:

  • BiGRU-Att & HAN: Recurrent models using hierarchical attention.
  • LWAN (Label-Wise Attention Network): A model designed for multi-label classification using per-label attention mechanisms.

Experimental Results

Binary Violation Prediction

HIER-BERT significantly outperformed traditional models, achieving an F1 score of 82.0 compared to the 71.8 achieved by the SVM baseline. A critical finding emerged during anonymization tests: when names and locations were replaced with generic tags, HIER-BERT’s performance dropped by approximately 2%. This suggests that models may inadvertently rely on demographic or geographic biases rather than purely legal facts.

Performance of various models on binary violation prediction for non-anonymized and anonymized datasets.

Multi-label and Importance Prediction

Predicting specific article violations (multi-label classification) proved more difficult than binary prediction. While HIER-BERT led with a 60.0 micro-F1, all models struggled with "few-shot" articles that rarely appeared in the training set. For case importance prediction, HIER-BERT achieved the highest Spearman’s correlation (0.527), indicating it is the most effective at ranking cases by their legal significance, even if absolute error remains a challenge due to the subjective nature of "importance."

Interpretability and Limitations

While neural models show high predictive accuracy, they remain "black boxes." Visualization of attention scores reveals that the models often focus on legally relevant terminology—such as "concussion," "beaten," or "hospital"—when predicting violations of articles related to ill-treatment.

Visualization of hierarchical attention scores highlighting key factual terms used for prediction.

Despite these promising results, attention maps are not equivalent to legal justifications. Future research will focus on incorporating external legal knowledge, testing across different jurisdictions (such as the US Supreme Court), and developing models capable of providing human-readable legal reasoning.

Summary updated

Abstract

Legal judgment prediction is the task of automatically predicting the outcome of a court case, given a text describing the case's facts. Previous work on using neural models for this task has focused on Chinese; only feature-based models (e.g., using bags of words and topics) have been considered in English. We release a new English legal judgment prediction dataset, containing cases from the European Court of Human Rights. We evaluate a broad variety of neural models on the new dataset, establishing strong baselines that surpass previous feature-based models in three tasks: (1) binary violation classification; (2) multi-label classification; (3) case importance prediction. We also explore if models are biased towards demographic information via data anonymization. As a side-product, we propose a hierarchical version of BERT, which bypasses BERT's length limitation.
BibTeX
@inproceedings{chalkidis2019neural,
  author = {Ilias Chalkidis and Ion Androutsopoulos and Nikolaos Aletras},
  title = {{Neural Legal Judgment Prediction in English}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=rsFm818Ku1q},
}
Figure