Accurate and Regret-aware Numerical Problem Solver for Tabular Question Answering

Yuxiang Wang, Jianzhong Qi, Junhao Gan

Abstract

Large Language Models (LLMs) often struggle with precise numerical reasoning in tabular contexts, frequently hallucinating arithmetic results. This paper introduces TabLaP, a framework that treats LLMs as code planners rather than calculators. By generating Python scripts for numerical tasks and incorporating a "regret-aware" trustworthiness evaluator, TabLaP significantly improves accuracy and helps users identify potentially incorrect outputs.

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in parsing unstructured and semi-structured data, such as web tables. However, they face a significant hurdle: numerical reasoning. When tasked with performing calculations—such as averaging values or summing columns—LLMs frequently fail at basic arithmetic. Furthermore, existing Table Question Answering (TableQA) systems rarely provide a measure of reliability, leading to "user regret" when incorrect answers are blindly trusted. TabLaP addresses these issues by combining code execution with a meta-evaluation layer.

The TabLaP Framework

To address these challenges, the authors propose TabLaP, a multi-LLM, dual-branch architecture designed for both accuracy and trustworthiness.

The TabLaP architecture consists of four stages: Input Data, Answer Generation (Dual-Branch), Answer Selection, and Trustworthiness Evaluation.

Dual-Branch Answer Generation

The system processes questions through two distinct paths:

  1. Branch A (SOTA): Utilizes a standard state-of-the-art TableQA model (Mix-SC) to handle general semantic and text-based queries.
  2. Branch B (NumSolver): A specialized module where an LLM (GPT-3.5 Turbo) acts as a planner. Instead of calculating the answer directly, it generates a Python script.

NumSolver: LLM as a Code Planner

The core innovation of NumSolver is the separation of planning and execution. The LLM describes the logical steps required to solve the problem in Python code, which is then executed by a dedicated interpreter. This ensures that the final result is mathematically precise, bypassing the LLM's inherent arithmetic limitations. If the code fails to execute, the system includes a self-correction fallback to direct reasoning.

NumSolver uses an LLM to generate Python code, which is then executed to ensure mathematical accuracy.

Answer Selection and Trustworthiness Evaluation

After both branches generate potential answers, the AnsSelector—a fine-tuned Llama 3-8B model—acts as a judge to determine which reasoning process is more likely to be correct based on the question and table schema.

Finally, the TwEvaluator (Trustworthiness Evaluator) quantifies the reliability of the chosen answer. It employs a Multi-Arm Bandit (MAB) algorithm to track the historical success rates of the different branches. By doing so, it attaches a "trustworthiness label" to the output, allowing users to gauge whether they should rely on the result or if the model is likely in a state of error.

Experimental Results

TabLaP was evaluated on several benchmarks, including WikiTableQuestions (WTQ) and a newly refined Fact-based Table QA (FTQ) dataset.

  • Accuracy Improvements: TabLaP achieved a 5.7% accuracy boost on WTQ and a 5.8% boost on FTQ over the previous state-of-the-art (Mix-SC), reaching overall accuracies of 76.6% and 44.1% respectively.
  • Numerical Precision: On questions requiring specific numerical reasoning, TabLaP outperformed baselines by over 6%.
  • Regret Reduction: The trustworthiness labels successfully reduced the "user regret ratio" by approximately 20% across both major datasets, demonstrating the system's ability to identify its own mistakes.

Conclusion

TabLaP demonstrates that the "notorious issue" of LLM math errors can be mitigated by shifting the model's role from calculator to architect. By combining code-based execution with a regret-aware evaluation mechanism, the framework provides a more robust and reliable solution for real-world tabular data analysis.

Summary updated

BibTeX
@inproceedings{wang2025accurate,
  author = {Yuxiang Wang and Jianzhong Qi and Junhao Gan},
  title = {{Accurate and Regret-aware Numerical Problem Solver for Tabular Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i12.33393},
}
Figure