Neural Legal Judgment Prediction in English

Ilias Chalkidis, Ion Androutsopoulos, Nikolaos Aletras

Abstract

Ilias Chalkidis, Ion Androutsopoulos, Nikolaos Aletras. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

2019.

Legal Judgment Prediction (LJP) is the task of automatically determining the outcome of a court case based on a description of its facts. While this field has seen significant progress in Chinese law, English-language research was previously limited by "shallow" feature-based models and small datasets. This paper introduces the first large-scale English dataset for LJP based on cases from the European Court of Human Rights (ECHR) and proposes HIER-BERT, a hierarchical version of BERT designed to handle long legal documents. HIER-BERT significantly outperforms traditional baselines and sets a new benchmark for predicting court outcomes and case importance.

Introduction

The challenge of applying modern NLP to English legal text is twofold. First, there is a significant data scarcity; until now, there was no large, publicly available English dataset containing raw text for neural model training. Second, document length poses a technical hurdle. Legal documents are notoriously long, often exceeding 2,000 words, while standard state-of-the-art models like BERT have a hard limit of 512 tokens. Truncating these documents results in the loss of critical legal context.

The ECHR Dataset

The authors introduced a new ECHR Dataset containing approximately 11.5k cases from the European Court of Human Rights. Unlike previous datasets that provided only pre-computed features, this dataset provides raw text, enabling the use of modern neural techniques. Each case includes a list of facts, the specific articles of the Convention violated (if any), and an "importance score" (1 to 4) assigned by the court.

Table 1: Statistics of the ECHR dataset subsets, showing case counts and average lengths.

Methods

The study evaluated several neural architectures to determine their efficacy in handling legal nuances:

  • BiGRU-Att: A bidirectional Gated Recurrent Unit with self-attention treating the case as one sequence.
  • HAN (Hierarchical Attention Network): This model mirrors legal structure, using one level of attention to encode individual "facts" and a second level to aggregate them into a case representation.
  • HIER-BERT: To bypass BERT’s length limitation, the authors used BERT to generate embeddings for individual paragraphs (facts), then applied a self-attention mechanism over those embeddings. This allows the model to process the entire case without truncation.
  • LWAN (Label-Wise Attention Network): Designed for multi-label tasks, this model uses multiple attention heads to seek evidence for specific legal articles.

Key Results and Analysis

Experimental results demonstrate that hierarchical models (HAN and HIER-BERT) consistently outperform flat models, suggesting that the inherent structure of legal documents—where a case is a collection of distinct facts—is a vital signal for AI.

Table 2: Performance comparison (Precision, Recall, F1) across models for binary violation prediction on non-anonymized and anonymized data.

Binary and Multi-label Performance

In binary violation prediction, HIER-BERT achieved an F1-score of 82.0, significantly beating the previous state-of-the-art SVM baseline of 71.8. Predicting specific article violations (multi-label classification) proved more difficult, though HIER-BERT still led with a 60.0 micro-F1. However, performance dropped sharply on "rare" articles, highlighting a challenge for few-shot learning in the legal domain.

Demographic Bias and Anonymization

By testing on "anonymized" data (replacing names and locations with generic tags), the authors found that models still performed well. However, HIER-BERT showed a slight performance drop when anonymized, suggesting it may be more prone to overfitting on specific demographic or geographic markers found in the training data. Furthermore, while attention maps highlighted relevant legal terms like "concussion," they also flagged specific locations, warning that models might reach correct conclusions for biased reasons.

Conclusion

While HIER-BERT sets a new state-of-the-art for English LJP, the authors emphasize that these models remain "black boxes." Attention scores provide hints but are not formal legal justifications. Future work should focus on expanding scope to other courts (e.g., the US Supreme Court), improving explainability, and incorporating external resources like prior judgments and case law to better simulate professional legal reasoning.

Summary updated

BibTeX
@inproceedings{chalkidis2019neural,
  author = {Ilias Chalkidis and Ion Androutsopoulos and Nikolaos Aletras},
  title = {{Neural Legal Judgment Prediction in English}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=rsFm818Ku1q},
}
Figure