Accurate and Regret-Aware Numerical Problem Solver for Tabular Question Answering

Yuxiang Wang, Jianzhong Qi, Junhao Gan

Introduction

Researchers from the University of Melbourne have introduced TabLaP (Table Language Planner), a multi-LLM framework designed to bridge the "math gap" in Tabular Question Answering (TableQA). While Large Language Models (LLMs) excel at semantic understanding, they frequently stumble over arithmetic operations, leading to hallucinations and user distrust. TabLaP addresses this by treating LLMs as strategic planners that offload numerical execution to a Python interpreter, while simultaneously providing a "trustworthiness" evaluator to make the system regret-aware.

The Numerical Bottleneck in TableQA

TableQA involves extracting and processing information from structured data, such as Wikipedia tables. Current state-of-the-art models often fail when a question requires multi-step arithmetic (e.g., calculating an average). Because LLMs predict the next token based on probability rather than logic, they may generate a plausible-looking but mathematically incorrect answer.

Comparison between a standard LLM's arithmetic failure and the successful execution of the LLM + Python Interpreter approach.

As shown in the comparison above, a standard LLM might miscalculate a total or average, whereas the integration of a Python interpreter ensures mathematical precision by executing generated code.

The TabLaP Framework

TabLaP operates through a three-stage pipeline designed to maximize both accuracy and reliability:

1. Dual-Branch Answer Generation

  • NumSolver: This module prompts the LLM to generate a Python script rather than a direct answer. This script is executed by a local interpreter to perform calculations. If the script fails, the system falls back to the LLM's internal reasoning.
  • SOTA Branch: For non-numerical or purely semantic queries, TabLaP utilizes a standard high-performing TableQA model (such as Mix-SC).
A sample Python script generated by the NumSolver module to calculate average distances accurately.

2. AnsSelector

To determine which branch provided the superior answer, a fine-tuned Llama 3-8B model acts as a judge. It evaluates the question, the table schema, and the outputs from both the NumSolver and the SOTA branch to select the final response.

3. TwEvaluator (Trustworthiness Evaluation)

This component introduces "regret-awareness." Using a Multi-Arm Bandit (MAB) algorithm, the TwEvaluator tracks the historical performance of different branches. It generates a trustworthiness label (True/False) alongside the answer, alerting the user if the model is operating in a high-uncertainty zone.

Experimental Results

TabLaP was evaluated on three major benchmarks: WikiTableQuestions (WTQ), FTQ, and TabFact. The results demonstrate a significant leap over previous state-of-the-art (SOTA) methods:

  • Accuracy Boost: TabLaP improved overall accuracy by 5.7% on WTQ and 5.8% on FTQ compared to previous leaders like Mix-SC.
  • Numerical Superiority: On questions specifically requiring arithmetic, TabLaP outperformed baselines by up to 6.8%.
  • Reduced Regret: The trustworthiness labels successfully reduced the "user regret ratio" by approximately 20%, meaning users were significantly less likely to be misled by incorrect model outputs.

Conclusion

By decoupling reasoning from calculation and introducing a dynamic trustworthiness evaluator, TabLaP provides a robust solution to the numerical limitations of LLMs in structured data tasks. Future work will focus on expanding the framework to handle more complex relational databases and refining the rejection filters for even greater reliability.

Summary updated

Abstract

Question answering on free-form tables (a.k.a. TableQA) is a challenging task because of the flexible structure and complex schema of tables. Recent studies use Large Language Models (LLMs) for this task, exploiting their capability in understanding the questions and tabular data, which are typically given in natural language and contain many textual fields, respectively. While this approach has shown promising results, it overlooks the challenges brought by numerical values which are common in tabular data, and LLMs are known to struggle with such values. We aim to address this issue, and we propose a model named TabLaP that uses LLMs as a planner rather than an answer generator. This approach exploits LLMs' capability in multi-step reasoning while leaving the actual numerical calculations to a Python interpreter for accurate calculation. Recognizing the inaccurate nature of LLMs, we further make a first attempt to quantify the trustworthiness of the answers produced by TabLaP, such that users can use TabLaP in a regret-aware manner. Experimental results on two benchmark datasets show that TabLaP is substantially more accurate than the state-of-the-art models, improving the answer accuracy by 5.7% and 5.8% on the two datasets, respectively.
BibTeX
@inproceedings{wang2025accurate,
  author = {Yuxiang Wang and Jianzhong Qi and Junhao Gan},
  title = {{Accurate and Regret-Aware Numerical Problem Solver for Tabular Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i12.33393},
}
Figure