Retrieval Model and Classification Model for AILA2020

Menghan Wu, Zhengyu Wu, Xiangyu Wang, Zhongyuan HanFIRE (Working Notes) 2020 · OpenReview

Introduction

The Artificial Intelligence for Legal Assistance (AILA) 2020 challenge addresses the critical need for automated tools in the legal domain. Legal professionals often struggle with the vast volume of precedents and statutes relevant to a given case. This paper presents a multi-pronged approach to three key tasks: Precedent Retrieval (Task 1a), Statute Retrieval (Task 1b), and Rhetorical Role Labeling (Task 2).

Problem Statement

The complexity of legal research stems from three main bottlenecks:

  1. Precedent Retrieval: Identifying relevant past cases from a corpus of thousands.
  2. Statute Retrieval: Pinpointing specific legal sections (Acts) applicable to a case description.
  3. Rhetorical Role Labeling: Segmenting judgments into functional components (e.g., Facts, Rulings) to facilitate navigation.

Legal texts are characterized by dense, specialized language and data sparseness, making traditional keyword-based retrieval challenging.

Methodology

The authors developed tailored strategies for each task:

Task 1a: Precedent Retrieval

The primary model used was BM25, a ranking function that improves upon TF-IDF by incorporating document length normalization and term frequency saturation. Experiments also included Cosine Similarity and a hybrid approach using TF-IDF for query term extraction followed by BM25 re-ranking.

Task 1b: Statute Retrieval

Given the brevity of statute descriptions (averaging 200 words), the authors employed Language Models with smoothing to mitigate the zero-probability problem for missing terms. Both Jelinek-Mercer and Dirichlet smoothing techniques were evaluated.

Task 2: Rhetorical Role Labeling

For sentence classification, a traditional Logistic Regression baseline with TF-IDF features was compared against a fine-tuned BERT-Base model. BERT was selected for its ability to capture contextual relationships within legal narratives.

Results and Discussion

Retrieval Performance

In Task 1a, the BM25 model achieved its best performance with a MAP of 0.1294. The hybrid approach yielded the highest Precision@10 (0.09).

Evaluation results for Task 1a (Precedent Retrieval) across different experimental runs.

For Task 1b, Dirichlet smoothing proved superior to Jelinek-Mercer, achieving a MAP of 0.2139. A key insight was the sensitivity of the Jelinek-Mercer model to the smoothing parameter $\lambda$.

Performance of Jelinek-Mercer smoothing on Task 1b as $\lambda$ approaches 1.

Parameter Sensitivity

The authors observed that standard BM25 parameters ($k_1=1.2, b=0.75$) were suboptimal for legal corpora. Adjusting these to $k_1=1.0$ and $b=1.0$ led to significant gains in MAP. Similarly, in Task 1b, performance improved as $\lambda$ approached 1, indicating that a high degree of smoothing is necessary for short, sparse legal descriptions.

Classification Performance

BERT-Base consistently outperformed Logistic Regression, reaching an accuracy of 0.578 and a Macro F-Score of 0.405. The model's contextual awareness allowed it to better distinguish between roles like "Facts" and "Arguments." However, the authors noted that training beyond 5 epochs led to overfitting on the specialized dataset.

Conclusion

The study demonstrates the effectiveness of tuned traditional retrieval models and modern transformer-based architectures in the legal domain. While the results are promising, future work will explore larger models like BERT-Large, legal-specific tokenization, and more advanced retrieval techniques to further improve performance in complex legal assistance tasks.

Summary updated

BibTeX
@inproceedings{wu2020retrieval,
  author = {Menghan Wu and Zhengyu Wu and Xiangyu Wang and Zhongyuan Han},
  title = {{Retrieval Model and Classification Model for AILA2020}},
  booktitle = {FIRE (Working Notes)},
  year = {2020},
  url = {https://openreview.net/forum?id=fK8HBqXZDH},
}
Figure