TL;DR
NitiBench+ is a new Thai legal QA benchmark that addresses a critical flaw in legal AI evaluation: the fact that law changes over time. By providing a massive, versioned corpus of Thai statutes, the authors demonstrate that grading models against "today's law" for "yesterday's cases" significantly distorts performance scores, and they introduce a temporal leaderboard to track the true state of Thai legal reasoning in LLMs.
What Problem Does This Paper Address?
Most legal benchmarks treat the law as a static text. However, in practice, a court must decide a case based on the law in force on the day of the dispute. If a benchmark asks a model to predict a 2014 court decision but evaluates it against 2026 statutes, the model is set up to fail if the law has been amended or repealed in the interim.
The authors audited 156,650 decisions from four Thai courts and found that "law drift" is pervasive: up to 67% of Supreme Court decisions cite law that has since changed. Despite this, an audit of 53 existing legal NLP benchmarks revealed that none of them grade models based on the specific version of the law in force at the time of the case. NitiBench+ fills this gap by requiring models to reason with temporally accurate legal texts.
Methods
The researchers developed a pipeline to extract facts, reasoning, and citations directly from court decisions using high-end LLMs (Claude Opus and Fable), followed by human verification.
Key methodological innovations include:
- Temporal Resolution: Every "gold" citation in the benchmark is resolved to the exact version active on the ruling date.
- Versioned Retrieval Corpus: They built a corpus containing 394,943 dated texts for over 106,000 statute sections, allowing models to be tested as retrieval agents over "in-force" law.
- Bar-Standard Scoring: Instead of simple multiple-choice, answers are free-form and graded on a 0–10 scale using the official Thai Bar Council rubric. This grading is performed by a panel of LLM judges calibrated against human bar examiners.
- Drift Analysis: The benchmark categorizes items into "No-drift" (control), "Amended," and "Repealed" to measure exactly how much performance is lost when a model reads the wrong version of a law.
Datasets
- NitiBench+: 1,670 items and 4,071 sub-questions spanning 17 domains (e.g., Family & Succession, Criminal, Tax & Revenue).
- FourCorners Corpus: A unified corpus of dated Thai statutes and rulings used for retrieval.
- Validation & Test Splits: A public validation split (498 cases) and a private, held-out test split (500 cases) used for the official leaderboard.
Key Insights and Ideas
- The Cost of the "Wrong" Text: Reading the current text of an amended law drops a model's Bar-standard score by 0.25 points. If the law was repealed, the score drops by a massive 0.79 points.
- Rules vs. Wording: The performance penalty comes almost entirely from changes to legal rules (0.45 point drop). Mere wording changes that don't alter the legal consequence have zero impact on scores.
- Memory is Not Enough: Without a search tool, only five open-weight models could recall any Thai law at all. For most models, "knowledge" of Thai law from pre-training is essentially a guess.
- Tool Use is the Great Divider: The ability to use a retrieval tool effectively is more important for performance than the specific Thai-language post-training a model has received.
Key Results
The paper presents a leaderboard of 28 models evaluated in three settings: Parametric (memory alone), Gold-cited (law provided in prompt), and RAG In-force (retrieval agent).
- Top Performer: Claude Fable 5.1 leads the pack with a score of 7.78/10 in the RAG setting.
- Open-Weight Leader: DeepSeek V4.1 Flash is the best open-weight model at 6.39/10.
- Reasoning Gap: Even when the correct legal sections are handed to the models (Gold-cited), no open-weight model passes a score of 6.4. This indicates that the primary bottleneck is not just finding the law, but applying it to the facts.
- Hallucination Rates: In the parametric (memory) setting, some models hallucinate citations at alarming rates—Nemotron 3 Nano, for instance, cites non-existent laws or sections 45.9% of the time.
Limitations and Future Work
- Scope: The benchmark currently focuses on three high courts and 17 domains. It does not yet cover military courts, lower courts, or advisory rulings from government agencies.
- LLM Judges: While the LLM judges are calibrated, the authors acknowledge they cannot fully replace human experts, especially on borderline points of law.
- Post-training Challenges: The authors found that current Thai-specific post-trainings often reduce a model's ability to use retrieval tools. A major open problem is developing post-training techniques that teach legal reasoning without breaking the model's general agentic capabilities.
Summary updated