THUIR@AILA 2019: Information Retrieval Approaches for Identifying Relevant Precedents and Statutes

Yunqiu Shao, Ziyi YeFIRE (Working Notes) 2019 · OpenReview

Introduction

In Common Law systems, legal professionals face the daunting task of navigating vast repositories of digitized documents to find relevant primary sources. These sources primarily consist of precedents (prior court rulings) and statutes (written laws). The AILA (Artificial Intelligence for Legal Assistance) 2019 track was established to address the inefficiencies of manual search by posing two challenges: identifying relevant prior cases (Task 1) and identifying relevant statutes (Task 2) based on a narrative description of a legal situation.

The primary difficulty in these tasks is the "long-to-long" matching problem. Legal queries are typically complex narratives averaging over 500 words, while target case documents can exceed 3,000 words. Standard keyword-based search algorithms often struggle to capture the semantic depth required for such long-form retrieval.

Methodology

Researchers from Tsinghua University (THUIR) approached both tasks as ranking problems, evaluating several classic Information Retrieval (IR) models and their ensembles:

  • Language Model for Information Retrieval (LMIR): This model estimates the probability of a query being generated by a document, utilizing unigram and bigram versions with linear interpolation smoothing.
  • Vector Space Model (VSM): Documents and queries are represented as vectors using TF-IDF weights, with similarity calculated via cosine distance.
  • BM25 (Okapi): A standard probabilistic ranking function designed to handle term frequency and document length normalization.
  • Mixed-size Bigram Model (MBM): A variation that considers both individual words and word pairs to better capture contextual nuances.

Task-Specific Pre-processing

To mitigate the noise in long queries, the team employed specific heuristics and summarization techniques:

  • Task 1 (Precedents): The team observed that legal conflicts often center on the appeal process. They extracted "key sentences" containing the phrase "high court" and the surrounding context to focus the model on the core legal issues.
  • Task 2 (Statutes): The team used TextRank, an unsupervised summarization algorithm, to condense 500-word situation descriptions into 200-word summaries, reducing irrelevant information.
  • Combination Models (CM): Weighted linear combinations of different model scores (e.g., VSM + BM25) were used to leverage the complementary strengths of individual approaches.

Results and Analysis

The THUIR team achieved significant success in Task 2, securing the top three rankings in the competition. The results highlighted a stark contrast in model effectiveness between the two tasks.

Performance metrics for Task 1, Task 2, and post-competition analysis.

As shown in the table above, the Vector Space Model (VSM) was the top performer for statute retrieval (Task 2), achieving a P@10 of 0.0975 and a MAP of 0.1566. Conversely, VSM was the worst-performing individual model for precedent retrieval (Task 1), where the Combination Model (CM) proved more robust, ranking 10th overall. This suggests that statutes may be more keyword-dependent, whereas precedents require more sophisticated semantic or structural matching.

Post-Competition Refinement

The authors identified a parameter error in their official Task 2 submission, where a weight was incorrectly set to 0.3 instead of 0.7. Post-competition experiments (labeled CM* in the results table) demonstrated that correcting this error significantly improved performance, with the Mean Average Precision (MAP) increasing from 0.1318 to 0.1742.

Discussion and Limitations

While the summarization and heuristic-based approaches were effective, the study noted several limitations. The small training set (only 10 labeled queries) made models susceptible to overfitting on specific keywords. Furthermore, the team intentionally avoided neural models due to the data scarcity, as deep learning typically requires larger datasets to outperform classic IR techniques.

The relatively lower performance in Task 1 underscores the complexity of "situation-to-case" matching. Future work will focus on more advanced modeling of the relationships between legal narratives and precedents, moving beyond traditional keyword-based retrieval to better handle the linguistic intricacies of court opinions.

Conclusion

The THUIR@AILA 2019 participation demonstrated that classic IR models, when combined with strategic pre-processing like TextRank and targeted text extraction, remain highly competitive in specialized domains like legal retrieval. Their success in Task 2 provides a strong baseline for future automated legal assistance systems.

Summary updated

BibTeX
@inproceedings{shao2019thuir,
  author = {Yunqiu Shao and Ziyi Ye},
  title = {{THUIR@AILA 2019: Information Retrieval Approaches for Identifying Relevant Precedents and Statutes}},
  booktitle = {FIRE (Working Notes)},
  year = {2019},
  url = {https://openreview.net/forum?id=8ixjGzeDR8},
}
Figure