Abstract
This contribution reports on an experiment on human label variation in annotating data for aligning legal case outcome prediction systems with experts in the context of the European Court of Human Rights (ECHR). With it, we seek to give a view into the "engine room" of the interdisciplinary development of software that uses natural language processing (NLP) to analyze court data in order to make predictions on case outcomes based on aspects of the case that lawyers consider relevant. While prior work in case outcome classification has been limited to annotations by a single expert, it is well-known that lawyers may disagree in their assessment of case facts. In high-stakes domains, like legal NLP, it is paramount for models to adequately capture the pluralism of human opinions and values. To this end, we construct a taxonomy of different sources of disagreement specific to annotation of legal texts and release a novel dataset of annotations of case fact aspects that are deemed relevant for determining violations of certain provisions of the European Convention on Human Rights. We obtained these annotations from two human rights law experts and explored the extent of their agreement using concepts from the taxonomy. Our case study reveals hitherto underappreciated complexities in creating benchmark datasets in this field that revolve around identifying aspects of a case's facts supposedly relevant for its outcome. This view into the engine room aims to illustrate how NLP research is conducted in the legal domain, and shows how modeling assumptions and human disagreement have a bearing on the reliability, explainability and transparency of the models.
Introduction: Beyond Simple Classification
Building AI to predict legal outcomes is relatively straightforward as a classification task; however, building AI that understands why a case was decided is significantly more complex. Most research in Legal Judgment Prediction (LJP) treats the law as a binary task: input the facts, output a "violation" or "no violation" label. Yet, these models often rely on "shallow signals"—such as the name of the defendant state or specific document structures—rather than actual legal reasoning.
To move toward meaningful decision-support tools, we must look into the "engine room" of legal NLP. This requires shifting focus from simple labels to the underlying rationales. A critical challenge in this transition is the inherent subjectivity of law. If two experts agree on a verdict but disagree on which facts were the "smoking guns," an AI trained on a single perspective will fail to capture the necessary pluralism of legal interpretation.
Methodology: The Taxonomy of Disagreement
Our research utilized an interdisciplinary approach, combining legal expertise with computer science and linguistics to study how experts identify relevant case facts. We developed a multi-level framework to categorize the sources of expert disagreement:
- Macro-Level Factors: Distinguishing between disagreements rooted in the data itself versus those rooted in the annotator's perspective.
- Fine-Grained Factors: Identifying specific causes such as Normative Uncertainty (vague legal provisions) or Narrative Unclarity (ambiguous or messy factual descriptions).
- Proxy Variables: To quantify these qualitative categories, we mapped them to court metadata, such as "Judge Split" (non-unanimous decisions) or "Different Prediction" (instances where the annotators themselves disagreed on the likely outcome).
For the annotation process, two human rights law experts independently reviewed 50 ECHR cases. Unlike traditional methods that mark entire paragraphs, our experts worked at the word and phrase level to identify specific "rationales"—the key snippets driving a legal conclusion.
The RaVE Dataset and the Metadata Trap
A primary contribution of this work is the RaVE (Rationale Variation in ECHR) dataset. This dataset contains dual-expert annotations for 50 cases from 2019–2022, specifically designed to study variation in legal reasoning.
Our analysis of existing data sources, specifically the official HUDOC database, revealed significant reliability issues. Approximately 25% of the metadata regarding alleged article violations was found to be incorrect or incomplete when verified by experts. This highlights a "metadata trap": researchers who treat court databases as absolute ground truth risk a "garbage in, garbage out" scenario where models learn from flawed labels.
Key Insights and Results
The "Judge Split" Paradox
Counterintuitively, our study found higher expert agreement on cases where the original ECHR judges were split. We hypothesize that these "milestone" cases often lead to more settled and clearer jurisprudence over time, making them easier for modern experts to annotate consistently compared to routine cases with less developed legal standards.
The Complexity of Article 6
Disagreement was highest in cases involving Article 6 (Right to a Fair Trial). The multi-faceted nature of this article makes it difficult for experts to provide consistent annotations without highly granular sub-categories, suggesting that general legal NLP models may struggle significantly with complex procedural rights.
Weak Congruence and the Model-Expert Gap
Quantitative analysis using the Kappa metric showed only "weak" agreement between the two experts. Rather than a failure of the experts, this reflects the inherent subjectivity of legal interpretation. Furthermore, preliminary tests showed limited overlap between what modern NLP models (using Saliency Maps) consider "important" and what human experts highlight. This suggests that high-accuracy models may simply be "statistically lucky" rather than legally sound.
Conclusion and Future Directions
The RaVE study demonstrates that "noise" in legal data is often actually "meaningful variation." For AI to be reliable and transparent in the legal domain, it must be designed to accommodate human disagreement rather than smoothing it over.
Future work must address the scalability of expert annotation and the potential of Generative AI. While Large Language Models (LLMs) like GPT-4 can generate rationales, they must be rigorously evaluated against expert data like RaVE to ensure they are not "hallucinating" legal logic. Ultimately, the path to trustworthy legal AI lies in better open access to data and a commitment to capturing the interpretive pluralism of the law.
Summary updated