Abstract
Large Language Models (LLMs) often struggle with precise numerical reasoning in tabular contexts, frequently hallucinating arithmetic results. This paper introduces TabLaP, a framework that treats LLMs as code planners rather than calculators. By generating Python scripts for numerical tasks and incorporating a "regret-aware" trustworthiness evaluator, TabLaP significantly improves accuracy and helps users identify potentially incorrect outputs.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in parsing unstructured and semi-structured data, such as web tables. However, they face a significant hurdle: numerical reasoning. When tasked with performing calculations—such as averaging values or summing columns—LLMs frequently fail at basic arithmetic. Furthermore, existing Table Question Answering (TableQA) systems rarely provide a measure of reliability, leading to "user regret" when incorrect answers are blindly trusted. TabLaP addresses these issues by combining code execution with a meta-evaluation layer.
The TabLaP Framework
To address these challenges, the authors propose TabLaP, a multi-LLM, dual-branch architecture designed for both accuracy and trustworthiness.
Dual-Branch Answer Generation
The system processes questions through two distinct paths:
- Branch A (SOTA): Utilizes a standard state-of-the-art TableQA model (Mix-SC) to handle general semantic and text-based queries.
- Branch B (NumSolver): A specialized module where an LLM (GPT-3.5 Turbo) acts as a planner. Instead of calculating the answer directly, it generates a Python script.
NumSolver: LLM as a Code Planner
The core innovation of NumSolver is the separation of planning and execution. The LLM describes the logical steps required to solve the problem in Python code, which is then executed by a dedicated interpreter. This ensures that the final result is mathematically precise, bypassing the LLM's inherent arithmetic limitations. If the code fails to execute, the system includes a self-correction fallback to direct reasoning.
Answer Selection and Trustworthiness Evaluation
After both branches generate potential answers, the AnsSelector—a fine-tuned Llama 3-8B model—acts as a judge to determine which reasoning process is more likely to be correct based on the question and table schema.
Finally, the TwEvaluator (Trustworthiness Evaluator) quantifies the reliability of the chosen answer. It employs a Multi-Arm Bandit (MAB) algorithm to track the historical success rates of the different branches. By doing so, it attaches a "trustworthiness label" to the output, allowing users to gauge whether they should rely on the result or if the model is likely in a state of error.
Experimental Results
TabLaP was evaluated on several benchmarks, including WikiTableQuestions (WTQ) and a newly refined Fact-based Table QA (FTQ) dataset.
- Accuracy Improvements: TabLaP achieved a 5.7% accuracy boost on WTQ and a 5.8% boost on FTQ over the previous state-of-the-art (Mix-SC), reaching overall accuracies of 76.6% and 44.1% respectively.
- Numerical Precision: On questions requiring specific numerical reasoning, TabLaP outperformed baselines by over 6%.
- Regret Reduction: The trustworthiness labels successfully reduced the "user regret ratio" by approximately 20% across both major datasets, demonstrating the system's ability to identify its own mistakes.
Conclusion
TabLaP demonstrates that the "notorious issue" of LLM math errors can be mitigated by shifting the model's role from calculator to architect. By combining code-based execution with a regret-aware evaluation mechanism, the framework provides a more robust and reliable solution for real-world tabular data analysis.
Summary updated