Abstract
Verifying the facts alleged by the prosecutors before the trial requires the judges to retrieve evidence within the massive materials accompanied. Existing Legal AI applications often assume the facts are already determined and fail to notice the difficulty of reconstructing them. To build a practical Legal AI application and free the judges from the manually searching work, we introduce the task of Legal Evidence Retrieval, which aims at automatically retrieving the precise fact-related verbal evidence within a single case. We formulate the task in a dense retrieval paradigm, and jointly learn the contrastive representations and alignments between facts and evidence. To get rid of the tedious annotations, we construct an approximated positive vector for a given fact by aggregating a set of evidence from the same case. An entropy-based denoise technique is further applied to mitigate the impact of false positive samples. We train our models on tens of thousands of unlabeled cases and evaluate them on a labeled dataset containing 919 cases and 4,336 queries. Experimental results indicate that our approach is effective and outperforms other state-of-the-art representation and retrieval models.
Introduction
In the legal domain, judges are tasked with verifying every fact alleged by a prosecutor by meticulously searching through mountains of testimony from defendants, victims, and witnesses. This process is currently a grueling manual effort. While Legal AI has advanced in areas like judgment prediction, it often assumes facts are already "proven."
We introduce Legal Evidence Retrieval (LER), a task focused on automatically linking prosecuted facts to specific pieces of verbal testimony within a case. This task is challenging due to the semantic gap between formal, third-person prosecutorial summaries and verbose, first-person witness testimony. Furthermore, retrieval must occur at a fine-grained sentence level within dynamic search pools that change for every case.
Methodology: The SWAP Framework
To address LER without the need for expensive manual labeling, we propose SWAP (Structure-aWare contrastive learning with Approximate aggregated Positive). SWAP is an unsupervised dense retrieval framework that leverages the inherent structure of legal cases to generate training signals.
The core of SWAP involves three key components:
- Dropout Positive: A standard data augmentation where a sentence is passed through the encoder twice with different dropout masks to create two slightly different representations.
- Approximate Aggregated Positive (AP): Recognizing that true evidence for a fact resides within the same case file, the model creates a "pseudo-positive" vector by aggregating all evidence sentences in that case, weighted by their similarity to the query fact.
- Instance Denoising: To handle cases where facts lack supporting evidence or are redundant, an entropy-based denoising method down-weights noisy samples during training.
Experimental Setup
We introduced the Legal Evidence Retrieval Dataset (LERD) to benchmark this task:
- LERD-usp: An unsupervised set with over 300,000 fact queries and 11 million candidate pairs from 35,423 criminal cases.
- LERD-sup: A high-quality evaluation set annotated by 10 professional lawyers, containing 4,336 queries with fine-grained relevance scores (0-2).
Unlike document-level retrieval datasets, LERD focuses on sentence-level alignment within a single case context.
Results and Discussion
Experimental results demonstrate that SWAP significantly outperforms traditional and state-of-the-art unsupervised models. SWAP-RoBERTa achieved a Mean Average Precision (MAP) of 61.45 and a Mean Reciprocal Rank (MRR) of 71.25. In contrast, the keyword-based BM25 only managed a MAP of 39.03, highlighting the severe semantic gap between prosecutorial and witness language.
Key insights include:
- Case Structure as Supervision: The case boundary provides a natural signal for contrastive learning. Knowing the answer must be in the current case file allows the model to generate its own signals.
- Aggregation Stability: Weighted aggregation of evidence provides a more stable training signal than trying to sample a single "correct" sentence in an unsupervised manner.
- Transferability: Pre-training with SWAP improves the performance of supervised models like DPR, indicating it effectively captures legal semantics.
Conclusion
The LER task and the SWAP framework represent a significant step toward practical Legal AI that assists judges in the fact-finding process. While currently focused on criminal cases and verbal testimony, future work could extend this to complex documentary evidence in civil cases and address the challenges of contested facts and contradictory accounts.
Summary updated