Argumentation-Driven Evidence Association in Criminal Cases

Yefei Teng, Wenhan ChaoEMNLP 2021 · DOI

Introduction

In many legal systems, criminal judgment documents typically present evidence in a flat "collection form." This format lists witness statements, forensic reports, and physical evidence without explicitly linking them to the specific legal claims they support. Only a small fraction (approximately 5%) of documents utilize an "argumentation-driven form," where evidence is partitioned into subsets that justify specific conclusions, such as a defendant's criminal responsibility or the circumstances of an incident. This lack of structure hinders readability and logical transparency. This paper addresses this gap by introducing the task of evidence association: automatically grouping judicial evidence into non-overlapping subsets based on the legal claims they substantiate.

Methodology: Distance Metrics for Clustering

The authors frame evidence association as a clustering problem where the number of clusters (claims) varies by case. They employ Agglomerative Hierarchical Clustering (AHC), which iteratively merges evidence items based on a dissimilarity metric. The core contribution is the development of three distinct distance metrics to guide this process:

  1. Latent Distance ($dist_{latent}$): Utilizes supervised models (BERT or ESIM) to predict the probability that two pieces of evidence support the same claim based on semantic content.
  2. Explicit Distance ($dist_{explicit}$): Measures the Jensen-Shannon divergence between the relevance distributions of evidence items across a set of explicit claims. This captures functional similarity (e.g., two different documents both proving identity).
  3. Ensemble Distance ($dist_{ensemble}$): A weighted combination of Latent and Explicit distances, balancing raw semantic similarity with claim-based functional inference.
Illustration of the three distance metrics (Latent, Explicit, and Ensemble) used for evidence clustering.

Dataset and Experimental Setup

The study utilizes a novel dataset curated from China Judgements Online, focusing on 500 cases of Traffic Accident Crime. These cases were selected because they were already written in the "argumentation-driven form," providing ground-truth labels for evidence-to-claim mappings. On average, cases contained 16.2 pieces of evidence and 11.9 claims. Notably, claim descriptions (avg. 45.5 characters) were significantly longer than evidence descriptions (avg. 10.7 characters), presenting a challenge for traditional short-text processing.

Results and Analysis

The models were evaluated using the Adjusted Rand Index (ARI) and Adjusted Mutual Information (AMI). Unsupervised baselines using GloVe embeddings and cosine distance performed poorly, highlighting the limitations of simple semantic similarity in legal contexts.

Performance comparison of unsupervised and supervised methods across ARI and AMI metrics.

As shown in the results table, supervised methods significantly outperformed the baseline. The BERT-based ensemble model achieved the highest performance with an ARI of 0.643 and an AMI of 0.656. These results demonstrate that combining semantic features with claim-relevance distributions effectively captures the nuances of legal argumentation.

Conclusion

This research demonstrates that evidence association can be effectively automated using supervised metric learning and hierarchical clustering. While the current approach relies on explicit claims, it provides a robust foundation for improving the interpretability of legal documents. Future work will explore modeling claims directly from latent fact descriptions to extend the system's utility to the 95% of cases currently lacking structured argumentation.

Summary updated

BibTeX
@inproceedings{teng2021argumentation,
  author = {Yefei Teng and Wenhan Chao},
  title = {{Argumentation-Driven Evidence Association in Criminal Cases}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.257},
}
Figure