NitiBench: Benchmarking LLM Frameworks on Thai Legal Question Answering Capabilities

Pawitsapak Akarajaradwong, Pirat Pothavorn, Chompakorn Chaksangchaichot, Panuthep Tasawong, Thitiwat Nopparatbundit, Keerakiat Pratai, Sarana NutanongEMNLP 2025

TL;DR

This paper introduces NitiBench, the first benchmark for Thai legal question answering (QA), addressing the critical need for robust LLM evaluation in underrepresented languages. By developing novel datasets and metrics for Thai financial and tax law, they reveal current limitations in LLM reasoning and retrieval, paving the way for more reliable legal AI.

What Problem Does This Paper Address?

Large Language Models (LLMs) show immense potential for legal question answering (QA), but their application in Thai legal contexts is hampered by a lack of specialized datasets and evaluation benchmarks. Existing English-language benchmarks and LLM frameworks struggle with the unique complexities of Thai legal structures, statutory retrieval, and the nuances of legal reasoning. Furthermore, practical Thai legal QA systems face limitations like strict API rate limits and inconsistent statute citation, making large-scale evaluation difficult. This paper aims to bridge this gap by creating a comprehensive benchmark for Thai legal QA.

Methods

The authors introduce NitiBench, a benchmark comprising two novel Thai legal QA datasets: NitiBench-CCL (Corporate and Commercial Law) and NitiBench-Tax (Tax Law). They also propose specialized multi-label retrieval metrics (e.g., Multi-HitRate@k, Multi-MRR@k) to better capture the nature of legal document retrieval, which often requires identifying multiple relevant sections. For end-to-end (E2E) evaluation, they use metrics like Coverage, Citation, and Contradiction, employing an LLM-as-a-judge approach.

Key methodological contributions include:

  • Hierarchy-aware Chunking: A novel chunking strategy that preserves the hierarchical structure of Thai legislation, aiming to improve retrieval by maintaining contextual integrity.
  • NitiLink: A framework that recursively fetches and incorporates inter-section references into the LLM context, addressing the challenge of linked legal texts.
  • LLM-as-a-Judge: Utilizing GPT-40 for evaluating citation precision, recall, F1, and contradiction, with prompts tuned for high human agreement.

Datasets

The paper introduces two new datasets:

  • NitiBench-CCL: Covers general financial law, derived from the WangchanX-Legal-ThaiCCL-RAG dataset, with a focus on simplifying LLM evaluation by removing rationales.
  • NitiBench-Tax: Specifically focuses on complex tax cases from 2021-2024, scraped from the Thai Revenue Department's website. This dataset is multi-labeled and designed to test nuanced legal reasoning.

Key Insights and Ideas

  • Hierarchy matters: Domain-specific chunking strategies that respect the hierarchical structure of Thai laws offer a slight but consistent advantage over naive chunking, suggesting that understanding legal structure is crucial for retrieval.
  • Retrieval is a bottleneck: Even with advanced LLMs, the performance of retrieval models significantly constrains end-to-end QA, especially for complex legal queries.
  • Long-context LLMs (LCLMs) lag behind RAG: While LCLMs can ingest vast amounts of text, they currently underperform RAG systems in Thai legal QA, particularly in tasks requiring precise citation and complex reasoning. This gap may stem from challenges in processing extremely long contexts and potential degradation in performance.
  • Multi-label metrics are essential: Traditional retrieval metrics are insufficient for legal QA. Multi-label metrics like Multi-MRR and Multi-HitRate show a stronger correlation with E2E performance, highlighting the need to evaluate retrieval of multiple relevant documents.

Key Results

  • Hierarchy-aware chunking slightly improved retrieval performance and end-to-end metrics compared to naive chunking on NitiBench-CCL.
  • NitiLink improved retriever recall but did not consistently translate to better E2E performance, with some cases showing performance decline due to increased context length.
  • Human-Finetuned BGE-M3 (HF BGE-M3) generally performed best among the evaluated retrievers on NitiBench-CCL.
  • Claude 3.5 Sonnet emerged as the top-performing LLM for Thai Legal QA, showing strong E2E recall and coverage on both datasets.
  • LCLM (Gemini 1.5 Pro) underperformed RAG systems on both datasets, with a significant drop in E2E F1 score as context depth increased.
  • The NitiBench-Tax dataset proved significantly more challenging, with larger recall gaps and higher hallucination rates across models, indicating the difficulty of complex legal reasoning in tax law.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Ambiguity: Queries in NitiBench-CCL, derived from single-section sampling, can be too general and overlap with multiple sections, potentially confusing LLMs.
  • Lack of Truly Multi-label Queries: While NitiBench-Tax addresses this, the broader dataset could benefit from more queries explicitly requiring multi-label reasoning.
  • Unnatural Phrasing: Queries in the datasets may not fully reflect typical user phrasing in Thai legal QA systems.
  • Limited Retriever Diversity: The evaluation of multi-label metrics was based on a limited number of retrievers due to cost and availability, impacting generalizability.
  • Legal Reasoning Evaluation: The current work focuses on the final answer and lacks a robust measurement of the LLM's legal reasoning process itself.

Future work could involve expanding the set of retrievers, incorporating more domain-specific datasets, and developing more sophisticated methods for evaluating legal reasoning.

Summary updated

BibTeX
@inproceedings{akarajaradwong2025nitibench,
  author = {Pawitsapak Akarajaradwong and Pirat Pothavorn and Chompakorn Chaksangchaichot and Panuthep Tasawong and Thitiwat Nopparatbundit and Keerakiat Pratai and Sarana Nutanong},
  title = {{NitiBench: Benchmarking LLM Frameworks on Thai Legal Question Answering Capabilities}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure