Abstract
Statute retrieval is a critical component of legal AI, yet existing benchmarks primarily focus on professional legal queries that do not reflect how the general public seeks legal advice. This paper introduces STARD, the first Chinese statute retrieval dataset featuring 1,543 real-world queries from non-professionals. We identify a significant "semantic gap" between informal "life facts" and formal "legal facts." By implementing a rigorous three-step annotation framework and evaluating various retrieval models, we demonstrate that while traditional and domain-specific models struggle with layperson language, fine-tuning and Retrieval-Augmented Generation (RAG) can significantly bridge this gap, improving the accessibility of legal information.
Introduction: The Semantic Gap in Legal Search
Finding the specific law or article that applies to a given situation—known as statute retrieval—is the backbone of legal AI. However, most existing benchmarks (such as COLIEE or AILA) utilize queries derived from bar exams or high-court documents. These queries are precise, structured, and laden with legal terminology.
In contrast, the general public uses informal, ambiguous language and lacks specific legal references. For example, a user might ask, "What should I do if my neighbor's dog bites me?" rather than referencing specific tort liability statutes. This discrepancy creates a "semantic gap" that current AI models, often trained on professional legalese, struggle to navigate. STARD was developed to address this challenge by providing a dataset that mirrors actual legal reasoning for non-professional queries.
Methodology: Bridging Life Facts and Legal Facts
To ensure high-quality data that accurately reflects legal reasoning, the researchers developed a three-step annotation framework. This process is designed to "translate" everyday descriptions into the rigid structure of statutory law.
- Recall: Annotators identify the broad legal category (e.g., Civil Law) and narrow it down to specific subfields (e.g., Tort Liability).
- Query Decomposition: This layer transforms informal "life facts" (e.g., "pet bite") into formal "legal facts" (e.g., "damage caused by domestic animals").
- Filter: Using a "subsumption" method, annotators match the interpreted legal facts against the smallest possible subset of statutes that fully address the query.
The STARD Dataset
The STARD dataset contains 1,543 real-world query cases collected from the "12348 China Legal Service" website. These queries are matched against a corpus of 55,348 candidate statutory articles from official Chinese regulations. On average, each query is associated with 1.76 relevant articles, highlighting the complexity of real-world legal issues where multiple statutes may apply.
Experimental Evaluation and Results
The researchers evaluated a variety of retrieval baselines, including traditional lexical matching (BM25), open-domain pre-trained language models (PLMs) like Roberta, and legal-specific models like Lawformer and SAILER.
Key findings from the experiments include:
- The Difficulty of Non-Professional Queries: Models trained on professional data (like CAIL) performed poorly on STARD, confirming that layperson queries represent a unique challenge.
- Fine-Tuning is Essential: The best-performing model, Dense-STARD, which was fine-tuned specifically on the new dataset, achieved a Recall@100 of 0.9065. However, the MRR@10 remained relatively low at 0.4724, indicating that precisely ranking the most relevant article in the top results is still difficult.
- Traditional IR Resilience: In zero-shot settings, traditional methods like BM25 often outperformed sophisticated neural models, suggesting that general-domain embeddings often fail to capture specialized legal relationships.
Discussion: RAG as a "Level Up" for Legal LLMs
The study also explored the impact of STARD on Large Language Models (LLMs) through Retrieval-Augmented Generation (RAG). By using STARD as an external knowledge base for models like Baichuan, ChatGLM, and ChatGPT, the researchers observed significant boosts in accuracy for legal tasks. For instance, using STARD-retrieved articles improved ChatGPT's performance on Knowledge-Driven legal questions from 0.206 to 0.252. This proves that even advanced LLMs require a reliable "open-book" reference to provide accurate legal advice to the public.
Conclusion and Future Work
STARD represents a significant step toward making legal AI more useful for the general public. While the dataset is currently specific to the Chinese Civil Law system, the methodology provides a blueprint for other jurisdictions. Future work will focus on expanding the dataset's legal scope and developing models that can better handle the complex reasoning required to link informal life facts to statutory language.
Summary updated