TUSI: A Citation-Grounded Graph-Based Benchmark

Marzieh Karimi, Mehran Safayani, Alireza BasiriarXiv 2026 · arXiv

TL;DR

The authors introduce TUSI-Bench, the first structured benchmark for evaluating Retrieval-Augmented Generation (RAG) in the Persian legal domain. By integrating a multi-layered knowledge graph of Iranian statutes with real-world lawyer-client consultations and Bar exam questions, the researchers demonstrate that "structure-aware" retrieval—leveraging the relationships between legal articles—boosts retrieval recall by up to 8 percentage points.

What Problem Does This Paper Address?

Deploying Large Language Models (LLMs) in the legal domain is a high-stakes endeavor. General-purpose models are notorious for "hallucinations," often fabricating case law or misinterpreting statutes. While RAG can mitigate this by grounding answers in external documents, the legal field presents unique hurdles:

  1. Jurisdictional Specificity: Law is not universal. A benchmark grounded in U.S. law is useless for the Iranian civil law system.
  2. The Lexical Gap: Laypeople describe legal problems in narrative, everyday language, while statutes use formal, archaic terminology. This makes simple keyword matching (like BM25) highly ineffective.
  3. Relational Complexity: A legally sound answer rarely comes from a single paragraph; it often requires connecting a specific article to its expert interpretations and related Supreme Court rulings.
  4. Low-Resource Languages: Persian lacks high-quality, grounded legal datasets. Existing resources are often synthetic or lack explicit links to authoritative legal texts.

Methods

The researchers built a comprehensive Structured Legal Knowledge Layer consisting of 7,261 articles across 50 statutes.

Figure 1: Overview of the structured legal knowledge layer and the proposed TUSI-Bench. TUSI-Bench consists of two complementary subsets: TUSI-QA, a citation-grounded question answering set derived from real lawyer–client consultations; and TUSI-Bar, a multiple-choice question set derived from Iranian Bar Exam questions.

The Knowledge Graph (KG)

Instead of treating the law as a flat list of text chunks, they constructed a Knowledge Graph where nodes represent articles, expert interpretations, and judicial rulings. Edges represent explicit cross-references or links between an article and its doctrinal commentary.

Structure-Aware Retrieval

The authors proposed a baseline that moves beyond simple semantic similarity. When a user asks a question, the system:

  1. Retrieves the most semantically similar text chunks using dense embeddings.
  2. Expands the context by including the "one-hop" neighbors of those chunks in the Knowledge Graph.
  3. Feeds this expanded, structurally-relevant evidence to the LLM to generate a grounded response.

Datasets

TUSI-Bench introduces two primary evaluation tasks:

  • TUSI-QA: A set of 1,999 open-ended questions derived from real lawyer-client interactions. Each question is paired with a professional answer and explicit citations to 1–3 supporting legal articles.
  • TUSI-Bar: Approximately 500 four-option multiple-choice questions taken from actual Iranian Bar examinations over the last three years, designed to test rigorous legal reasoning.

The authors highlight that TUSI-QA features significantly longer and more complex queries and answers compared to general-domain Persian QA datasets like PersianQuAD.

Figure 2: Length distributions of statutory articles (left) and their corresponding legal interpretations (right).

Key Insights and Ideas

  • The Power of the One-Hop: Simply adding the immediate neighbors of a retrieved article in the Knowledge Graph significantly improves the system's ability to find the "missing pieces" of a legal argument that semantic search alone might overlook.
  • Lexical Similarity is a Trap: The paper finds that the lexical overlap between user queries and relevant laws in TUSI-QA is extremely low (0.03). This confirms that legal RAG cannot rely on surface-level text matching and requires deep semantic and structural understanding.
  • Grounding via Filtering: The authors used an iterative "human-in-the-loop" process with an AI agent and legal experts to filter their dataset. This ensured that every question in the benchmark is actually answerable using the provided statutory articles, creating a "gold standard" for citation-level evaluation.

Key Results

  • Retrieval Boost: Incorporating graph-based relations improved Recall@5 by up to 8 percentage points across various embedding models (e.g., Jina-v3, BGE-M3).
  • Downstream Reasoning: In the TUSI-QA task, the structure-aware RAG approach led to higher scores in Correctness, Effectiveness, and Completeness when evaluated by an LLM-as-a-judge (DeepSeek-V4-Pro).
  • Bar Exam Performance: The structure-aware approach improved accuracy on the TUSI-Bar multiple-choice task from a baseline of 33.0% to 37.7%. While an improvement, the low absolute scores underscore how challenging this benchmark is for current models.

Limitations and Future Work

  • Unweighted Graphs: Currently, the Knowledge Graph treats all relationships (edges) as equal. The authors suggest that weighting edges based on citation frequency or relation type could further refine retrieval.
  • Scalability of Curation: Building the multi-layered knowledge base requires significant expert effort. Automating the extraction of relations from legal text is a key area for future research.
  • Expanding the Corpus: The current benchmark covers 50 major statutes. Expanding this to include the full breadth of executive regulations and a larger volume of judicial precedents is the next logical step.

Summary updated

Abstract

Evaluating retrieval-augmented generation (RAG) systems in Persian law remains challenging because benchmarks lack realism, grounding, and coverage of legal reasoning. Existing resources are either synthetically generated from statutory texts, lacking natural information needs, or derived from informal legal question--answer sources without explicit linkage to authoritative legal documents, making them unsuitable for rigorous evaluation of grounded legal reasoning. To address this gap, we introduce TUSI-Bench, a structured Persian legal benchmark designed for the evaluation of RAG-based systems. TUSI-Bench is built upon a multi-layered legal knowledge framework containing 7,261 legal articles and integrating codified statutes, legal commentaries, doctrinal writings, and Supreme Court unification rulings. We also construct a legal knowledge graph linking articles and concepts for structure-aware retrieval. Based on this foundation, TUSI-Bench provides two complementary evaluation tasks: (i) TUSI-QA, an open-ended question answering benchmark containing 1,999 questions that requires grounded responses with explicit legal citations, and (ii) TUSI-Bar, a multiple-choice benchmark containing approximately 500 four-option questions derived from Iranian Bar examination questions. Furthermore, we propose a graph-based retrieval baseline that leverages relational structure within the legal knowledge graph. Experimental results demonstrate that incorporating graph-based relations improves retrieval recall by up to 8 percentage points compared to standard baselines, highlighting the importance of structured legal representations for RAG systems. The proposed benchmark provides a standardized resource for evaluating grounded legal reasoning in Persian and enables future research on structured legal intelligence.
BibTeX
@misc{karimi2026tusi,
  author = {Marzieh Karimi and Mehran Safayani and Alireza Basiri},
  title = {{TUSI: A Citation-Grounded Graph-Based Benchmark}},
  year = {2026},
  eprint = {2611.21823},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21823},
}
Figure