TL;DR
The authors introduce TUSI-Bench, the first structured benchmark for evaluating Retrieval-Augmented Generation (RAG) in the Persian legal domain. By integrating a multi-layered knowledge graph of Iranian statutes with real-world lawyer-client consultations and Bar exam questions, the researchers demonstrate that "structure-aware" retrieval—leveraging the relationships between legal articles—boosts retrieval recall by up to 8 percentage points.
What Problem Does This Paper Address?
Deploying Large Language Models (LLMs) in the legal domain is a high-stakes endeavor. General-purpose models are notorious for "hallucinations," often fabricating case law or misinterpreting statutes. While RAG can mitigate this by grounding answers in external documents, the legal field presents unique hurdles:
- Jurisdictional Specificity: Law is not universal. A benchmark grounded in U.S. law is useless for the Iranian civil law system.
- The Lexical Gap: Laypeople describe legal problems in narrative, everyday language, while statutes use formal, archaic terminology. This makes simple keyword matching (like BM25) highly ineffective.
- Relational Complexity: A legally sound answer rarely comes from a single paragraph; it often requires connecting a specific article to its expert interpretations and related Supreme Court rulings.
- Low-Resource Languages: Persian lacks high-quality, grounded legal datasets. Existing resources are often synthetic or lack explicit links to authoritative legal texts.
Methods
The researchers built a comprehensive Structured Legal Knowledge Layer consisting of 7,261 articles across 50 statutes.
The Knowledge Graph (KG)
Instead of treating the law as a flat list of text chunks, they constructed a Knowledge Graph where nodes represent articles, expert interpretations, and judicial rulings. Edges represent explicit cross-references or links between an article and its doctrinal commentary.
Structure-Aware Retrieval
The authors proposed a baseline that moves beyond simple semantic similarity. When a user asks a question, the system:
- Retrieves the most semantically similar text chunks using dense embeddings.
- Expands the context by including the "one-hop" neighbors of those chunks in the Knowledge Graph.
- Feeds this expanded, structurally-relevant evidence to the LLM to generate a grounded response.
Datasets
TUSI-Bench introduces two primary evaluation tasks:
- TUSI-QA: A set of 1,999 open-ended questions derived from real lawyer-client interactions. Each question is paired with a professional answer and explicit citations to 1–3 supporting legal articles.
- TUSI-Bar: Approximately 500 four-option multiple-choice questions taken from actual Iranian Bar examinations over the last three years, designed to test rigorous legal reasoning.
The authors highlight that TUSI-QA features significantly longer and more complex queries and answers compared to general-domain Persian QA datasets like PersianQuAD.
Key Insights and Ideas
- The Power of the One-Hop: Simply adding the immediate neighbors of a retrieved article in the Knowledge Graph significantly improves the system's ability to find the "missing pieces" of a legal argument that semantic search alone might overlook.
- Lexical Similarity is a Trap: The paper finds that the lexical overlap between user queries and relevant laws in TUSI-QA is extremely low (0.03). This confirms that legal RAG cannot rely on surface-level text matching and requires deep semantic and structural understanding.
- Grounding via Filtering: The authors used an iterative "human-in-the-loop" process with an AI agent and legal experts to filter their dataset. This ensured that every question in the benchmark is actually answerable using the provided statutory articles, creating a "gold standard" for citation-level evaluation.
Key Results
- Retrieval Boost: Incorporating graph-based relations improved Recall@5 by up to 8 percentage points across various embedding models (e.g., Jina-v3, BGE-M3).
- Downstream Reasoning: In the TUSI-QA task, the structure-aware RAG approach led to higher scores in Correctness, Effectiveness, and Completeness when evaluated by an LLM-as-a-judge (DeepSeek-V4-Pro).
- Bar Exam Performance: The structure-aware approach improved accuracy on the TUSI-Bar multiple-choice task from a baseline of 33.0% to 37.7%. While an improvement, the low absolute scores underscore how challenging this benchmark is for current models.
Limitations and Future Work
- Unweighted Graphs: Currently, the Knowledge Graph treats all relationships (edges) as equal. The authors suggest that weighting edges based on citation frequency or relation type could further refine retrieval.
- Scalability of Curation: Building the multi-layered knowledge base requires significant expert effort. Automating the extraction of relations from legal text is a key area for future research.
- Expanding the Corpus: The current benchmark covers 50 major statutes. Expanding this to include the full breadth of executive regulations and a larger volume of judicial precedents is the next logical step.
Summary updated