NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains

Pawitsapak Akarajaradwong, Sarana Nutanong, Chompakorn ChaksangchaichotarXiv 2026 · arXiv

TL;DR

NitiBench+ is a new Thai legal QA benchmark that addresses a critical flaw in legal AI evaluation: the fact that law changes over time. By providing a massive, versioned corpus of Thai statutes, the authors demonstrate that grading models against "today's law" for "yesterday's cases" significantly distorts performance scores, and they introduce a temporal leaderboard to track the true state of Thai legal reasoning in LLMs.

What Problem Does This Paper Address?

Most legal benchmarks treat the law as a static text. However, in practice, a court must decide a case based on the law in force on the day of the dispute. If a benchmark asks a model to predict a 2014 court decision but evaluates it against 2026 statutes, the model is set up to fail if the law has been amended or repealed in the interim.

The authors audited 156,650 decisions from four Thai courts and found that "law drift" is pervasive: up to 67% of Supreme Court decisions cite law that has since changed. Despite this, an audit of 53 existing legal NLP benchmarks revealed that none of them grade models based on the specific version of the law in force at the time of the case. NitiBench+ fills this gap by requiring models to reason with temporally accurate legal texts.

Figure 2. Drift by decision year, 1935–2025, stacked to 100%. Each decision takes the most severe state among its citations. Totals in Table 1.

Methods

The researchers developed a pipeline to extract facts, reasoning, and citations directly from court decisions using high-end LLMs (Claude Opus and Fable), followed by human verification.

Key methodological innovations include:

  • Temporal Resolution: Every "gold" citation in the benchmark is resolved to the exact version active on the ruling date.
  • Versioned Retrieval Corpus: They built a corpus containing 394,943 dated texts for over 106,000 statute sections, allowing models to be tested as retrieval agents over "in-force" law.
  • Bar-Standard Scoring: Instead of simple multiple-choice, answers are free-form and graded on a 0–10 scale using the official Thai Bar Council rubric. This grading is performed by a panel of LLM judges calibrated against human bar examiners.
  • Drift Analysis: The benchmark categorizes items into "No-drift" (control), "Amended," and "Repealed" to measure exactly how much performance is lost when a model reads the wrong version of a law.

Datasets

  • NitiBench+: 1,670 items and 4,071 sub-questions spanning 17 domains (e.g., Family & Succession, Criminal, Tax & Revenue).
  • FourCorners Corpus: A unified corpus of dated Thai statutes and rulings used for retrieval.
  • Validation & Test Splits: A public validation split (498 cases) and a private, held-out test split (500 cases) used for the official leaderboard.

Key Insights and Ideas

  • The Cost of the "Wrong" Text: Reading the current text of an amended law drops a model's Bar-standard score by 0.25 points. If the law was repealed, the score drops by a massive 0.79 points.
  • Rules vs. Wording: The performance penalty comes almost entirely from changes to legal rules (0.45 point drop). Mere wording changes that don't alter the legal consequence have zero impact on scores.
  • Memory is Not Enough: Without a search tool, only five open-weight models could recall any Thai law at all. For most models, "knowledge" of Thai law from pre-training is essentially a guess.
  • Tool Use is the Great Divider: The ability to use a retrieval tool effectively is more important for performance than the specific Thai-language post-training a model has received.
Table 1. Decisions whose cited law has changed since the decision, by court, in percent. A decision takes the most severe state among its citations; Drifted is Amended plus Repealed.

Key Results

The paper presents a leaderboard of 28 models evaluated in three settings: Parametric (memory alone), Gold-cited (law provided in prompt), and RAG In-force (retrieval agent).

  • Top Performer: Claude Fable 5.1 leads the pack with a score of 7.78/10 in the RAG setting.
  • Open-Weight Leader: DeepSeek V4.1 Flash is the best open-weight model at 6.39/10.
  • Reasoning Gap: Even when the correct legal sections are handed to the models (Gold-cited), no open-weight model passes a score of 6.4. This indicates that the primary bottleneck is not just finding the law, but applying it to the facts.
  • Hallucination Rates: In the parametric (memory) setting, some models hallucinate citations at alarming rates—Nemotron 3 Nano, for instance, cites non-existent laws or sections 45.9% of the time.

Limitations and Future Work

  • Scope: The benchmark currently focuses on three high courts and 17 domains. It does not yet cover military courts, lower courts, or advisory rulings from government agencies.
  • LLM Judges: While the LLM judges are calibrated, the authors acknowledge they cannot fully replace human experts, especially on borderline points of law.
  • Post-training Challenges: The authors found that current Thai-specific post-trainings often reduce a model's ability to use retrieval tools. A major open problem is developing post-training techniques that teach legal reasoning without breaking the model's general agentic capabilities.

Summary updated

Abstract

Law changes over time. A court decides a case under the law in force on that day, and legal QA benchmarks grade answers against today's law. We checked 156,650 decisions of four Thai courts. Up to 67% of them cite law that has since been amended or repealed. We introduce NitiBench+, the sequel of our NitiBench, a Thai legal QA benchmark with 1,670 items across 17 domains, built from decisions of three courts. For each cited section it holds both the text in force on the decision date and the current text, and its retrieval corpus holds every dated version of every Thai statute section. Answers are graded on the Thai Bar Council rubric, the Bar-standard score, and on the overlap between the cited sections and the court's. The report has two parts. Part I measures the cost of the wrong text on 13 open-weight models. Reading the current text of amended law drops the Bar-standard score by 0.25 and missing repealed law drops it by 0.79. The penalty comes from rule changes, 0.45 on the 428 rule-changed items and none on the 331 wording-only items, and Citation F1 does not see it. The text in force is the fair setting. Part II benchmarks today's models on that setting. A public leaderboard ranks 26 models on a held-out test split of 500 items stratified by domain, with a validation split of the same shape released for public use, from memory alone, with the court's sections provided, and as a retrieval agent over the law in force, with the cost per item and the hallucinated citations. The best model scores 7.78 out of 10 and the best open-weight model 6.39. Without a search tool only five open-weight models recall any Thai law, and with it twenty of the 26 models gain and six lose. Leaderboard will take new models as they are released. Legal benchmarks must track dates, and on the right text today's models still leave a wide gap on Thai law.
BibTeX
@misc{akarajaradwong2026nitibench,
  author = {Pawitsapak Akarajaradwong and Sarana Nutanong and Chompakorn Chaksangchaichot},
  title = {{NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains}},
  year = {2026},
  eprint = {2611.03576},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03576},
}
Figure