Diverse legal case search

Ruizhe Zhang, Qingyao Ai, Yueyue Wu, Yixiao Ma, Yiqun LiuarXiv 2023 · DOI · arXiv

In the specialized domain of legal research, the standard paradigm of relevance-based retrieval often falls short of practitioner needs. A lawyer defending a client does not merely seek the most identical past judgment; they require a comprehensive understanding of the legal landscape, including overlapping charges and related subtopics that might influence a final ruling.

Research from Tsinghua University, presented at WSDM '23, addresses this gap in the paper "Diverse legal case search." The authors argue that current legal retrieval systems fail by focusing exclusively on relevance while ignoring the critical requirement for diversity in search results.

Traditional search engines are optimized for relevance: a query for "A" should return the best matches for "A." In open-domain web search, diversity is typically used to resolve ambiguity (e.g., distinguishing between "Mercury" the planet and the element). However, in the legal context, diversity is an exploratory tool. Legal subtopics, or charges, are rarely independent; they are logically and legally interconnected.

To quantify this need, the researchers analyzed commercial search logs. Their findings revealed that legal practitioners frequently exhibit "diversified search intents."

Figure 2: Distribution of charges per session based on query terms, illustrating that nearly half of all sessions involve multiple legal intents.

As shown in the log analysis, approximately 46% of queries contained multiple charges, and 89% of sessions involved users clicking on documents spanning different legal charges. This empirical evidence underscores a significant demand for systems that can surface a variety of related legal concepts rather than just the most semantically similar text.

The authors proposed the Diversified Legal case Retrieval Model (DLRM), a framework designed to move beyond simple text similarity by modeling the complex relationships between legal charges using a knowledge graph.

Figure 4: The DLRM architecture, featuring parallel modules for text similarity and charge relationship modeling.

The DLRM architecture consists of three primary components:

  1. Text Similarity Module (C-SW): Legal documents are often too long for standard transformer models. DLRM employs a "Cut with Sliding Windows" (C-SW) approach, segmenting documents into overlapping passages and using BERT to generate semantic embeddings.
  2. Charge Similarity Module (RWoG): This core innovation utilizes a Legal Knowledge Graph where nodes represent charges and edges represent the frequency with which one charge is modified or reversed by another in court trials. A Random Walk on Graph (RWoG) algorithm learns embeddings that capture these logical legal relationships.
  3. Ranking Prediction (MLP): A Multi-Layer Perceptron integrates text features with charge relationship features to produce a final ranking score.

Dataset and Experimental Results

To evaluate their model, the authors developed the DLR-dataset, the first human-annotated legal search dataset specifically designed for diversity. It contains 106 criminal cases from the Chinese legal system, with candidate documents labeled by legal experts for relevance to specific subtopics.

Experimental results demonstrated that DLRM significantly outperforms state-of-the-art retrieval methods:

  • Superior Performance: DLRM exceeded all baselines across standard diversity metrics, including $\alpha$-NDCG and NDCG-IA.
  • Significant Gains: On the NDCG-IA@10 metric, DLRM showed a 37.3% improvement over classic BM25 and a 20% improvement over the best existing diversification baseline (exIA-select).
  • Ablation Insights: Interestingly, a "Charge Only" version of the model performed better than a "Text Only" version, suggesting that understanding legal concepts is often more critical for effective retrieval than simple keyword matching.

Conclusion and Future Directions

DLRM represents a significant advancement in legal AI by explicitly modeling the correlations between legal subtopics. While the current study focuses on the Chinese criminal legal system and a relatively small expert-labeled dataset, it provides a blueprint for future work. Expanding this approach to Common Law jurisdictions and developing evaluation metrics tailored specifically to legal practitioner satisfaction remain key areas for future exploration.

Summary updated

Abstract

In last decades, legal case search has received more and more attention. Legal practitioners need to work or enhance their efficiency by means of class case search. In the process of searching, legal practitioners often need the search results under several different causes of cases as reference. However, existing work tends to focus on the relevance of the judgments themselves, without considering the connection between the causes of action. Several well-established diversity search techniques already exist in open-field search efforts. However, these techniques do not take into account the specificity of legal search scenarios, e.g., the subtopic may not be independent of each other, but somehow connected. Therefore, we construct a diversity legal retrieval model. This model takes into account both diversity and relevance, and is well adapted to this scenario. At the same time, considering the lack of dataset with diversity labels, we constructed a diversity legal retrieval dataset and obtained labels by manual labeling. experiments confirmed that our model is effective.
BibTeX
@misc{zhang2023diverse,
  author = {Ruizhe Zhang and Qingyao Ai and Yueyue Wu and Yixiao Ma and Yiqun Liu},
  title = {{Diverse legal case search}},
  year = {2023},
  eprint = {2301.12504},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.12504},
}
Figure