STARD: A Chinese Statute Retrieval Dataset with Real Queries Issued by Non-professionals

Weihang Su, Yiran Hu, Anzhe Xie, Qingyao Ai, Zibing Que, Ning Zheng, Yun Liu, Weixing Shen, Yiqun LiuarXiv 2024 · DOI · arXiv

Abstract

Statute retrieval is a critical component of legal AI, yet existing benchmarks primarily focus on professional legal queries that do not reflect how the general public seeks legal advice. This paper introduces STARD, the first Chinese statute retrieval dataset featuring 1,543 real-world queries from non-professionals. We identify a significant "semantic gap" between informal "life facts" and formal "legal facts." By implementing a rigorous three-step annotation framework and evaluating various retrieval models, we demonstrate that while traditional and domain-specific models struggle with layperson language, fine-tuning and Retrieval-Augmented Generation (RAG) can significantly bridge this gap, improving the accessibility of legal information.

Finding the specific law or article that applies to a given situation—known as statute retrieval—is the backbone of legal AI. However, most existing benchmarks (such as COLIEE or AILA) utilize queries derived from bar exams or high-court documents. These queries are precise, structured, and laden with legal terminology.

In contrast, the general public uses informal, ambiguous language and lacks specific legal references. For example, a user might ask, "What should I do if my neighbor's dog bites me?" rather than referencing specific tort liability statutes. This discrepancy creates a "semantic gap" that current AI models, often trained on professional legalese, struggle to navigate. STARD was developed to address this challenge by providing a dataset that mirrors actual legal reasoning for non-professional queries.

To ensure high-quality data that accurately reflects legal reasoning, the researchers developed a three-step annotation framework. This process is designed to "translate" everyday descriptions into the rigid structure of statutory law.

  1. Recall: Annotators identify the broad legal category (e.g., Civil Law) and narrow it down to specific subfields (e.g., Tort Liability).
  2. Query Decomposition: This layer transforms informal "life facts" (e.g., "pet bite") into formal "legal facts" (e.g., "damage caused by domestic animals").
  3. Filter: Using a "subsumption" method, annotators match the interpreted legal facts against the smallest possible subset of statutes that fully address the query.
The three-step annotation framework used to construct the STARD dataset, mapping informal queries to specific legal articles.

The STARD Dataset

The STARD dataset contains 1,543 real-world query cases collected from the "12348 China Legal Service" website. These queries are matched against a corpus of 55,348 candidate statutory articles from official Chinese regulations. On average, each query is associated with 1.76 relevant articles, highlighting the complexity of real-world legal issues where multiple statutes may apply.

Experimental Evaluation and Results

The researchers evaluated a variety of retrieval baselines, including traditional lexical matching (BM25), open-domain pre-trained language models (PLMs) like Roberta, and legal-specific models like Lawformer and SAILER.

Retrieval performance of various models on the STARD dataset across multiple metrics, showing the superiority of fine-tuned models.

Key findings from the experiments include:

  • The Difficulty of Non-Professional Queries: Models trained on professional data (like CAIL) performed poorly on STARD, confirming that layperson queries represent a unique challenge.
  • Fine-Tuning is Essential: The best-performing model, Dense-STARD, which was fine-tuned specifically on the new dataset, achieved a Recall@100 of 0.9065. However, the MRR@10 remained relatively low at 0.4724, indicating that precisely ranking the most relevant article in the top results is still difficult.
  • Traditional IR Resilience: In zero-shot settings, traditional methods like BM25 often outperformed sophisticated neural models, suggesting that general-domain embeddings often fail to capture specialized legal relationships.

The study also explored the impact of STARD on Large Language Models (LLMs) through Retrieval-Augmented Generation (RAG). By using STARD as an external knowledge base for models like Baichuan, ChatGLM, and ChatGPT, the researchers observed significant boosts in accuracy for legal tasks. For instance, using STARD-retrieved articles improved ChatGPT's performance on Knowledge-Driven legal questions from 0.206 to 0.252. This proves that even advanced LLMs require a reliable "open-book" reference to provide accurate legal advice to the public.

Conclusion and Future Work

STARD represents a significant step toward making legal AI more useful for the general public. While the dataset is currently specific to the Chinese Civil Law system, the methodology provides a blueprint for other jurisdictions. Future work will focus on expanding the dataset's legal scope and developing models that can better handle the complex reasoning required to link informal life facts to statutory language.

Summary updated

Abstract

Statute retrieval aims to find relevant statutory articles for specific queries. This process is the basis of a wide range of legal applications such as legal advice, automated judicial decisions, legal document drafting, etc. Existing statute retrieval benchmarks focus on formal and professional queries from sources like bar exams and legal case documents, thereby neglecting non-professional queries from the general public, which often lack precise legal terminology and references. To address this gap, we introduce the STAtute Retrieval Dataset (STARD), a Chinese dataset comprising 1,543 query cases collected from real-world legal consultations and 55,348 candidate statutory articles. Unlike existing statute retrieval datasets, which primarily focus on professional legal queries, STARD captures the complexity and diversity of real queries from the general public. Through a comprehensive evaluation of various retrieval baselines, we reveal that existing retrieval approaches all fall short of these real queries issued by non-professional users. The best method only achieves a Recall@100 of 0.907, suggesting the necessity for further exploration and additional research in this area. All the codes and datasets are available at: https://github.com/oneal2000/STARD/tree/main
BibTeX
@misc{su2024stard,
  author = {Weihang Su and Yiran Hu and Anzhe Xie and Qingyao Ai and Zibing Que and Ning Zheng and Yun Liu and Weixing Shen and Yiqun Liu},
  title = {{STARD: A Chinese Statute Retrieval Dataset with Real Queries Issued by Non-professionals}},
  year = {2024},
  eprint = {2406.15313},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.15313},
}
Figure