Introduction
Researchers from the University of Melbourne have introduced TabLaP (Table Language Planner), a multi-LLM framework designed to bridge the "math gap" in Tabular Question Answering (TableQA). While Large Language Models (LLMs) excel at semantic understanding, they frequently stumble over arithmetic operations, leading to hallucinations and user distrust. TabLaP addresses this by treating LLMs as strategic planners that offload numerical execution to a Python interpreter, while simultaneously providing a "trustworthiness" evaluator to make the system regret-aware.
The Numerical Bottleneck in TableQA
TableQA involves extracting and processing information from structured data, such as Wikipedia tables. Current state-of-the-art models often fail when a question requires multi-step arithmetic (e.g., calculating an average). Because LLMs predict the next token based on probability rather than logic, they may generate a plausible-looking but mathematically incorrect answer.
As shown in the comparison above, a standard LLM might miscalculate a total or average, whereas the integration of a Python interpreter ensures mathematical precision by executing generated code.
The TabLaP Framework
TabLaP operates through a three-stage pipeline designed to maximize both accuracy and reliability:
1. Dual-Branch Answer Generation
- NumSolver: This module prompts the LLM to generate a Python script rather than a direct answer. This script is executed by a local interpreter to perform calculations. If the script fails, the system falls back to the LLM's internal reasoning.
- SOTA Branch: For non-numerical or purely semantic queries, TabLaP utilizes a standard high-performing TableQA model (such as Mix-SC).
2. AnsSelector
To determine which branch provided the superior answer, a fine-tuned Llama 3-8B model acts as a judge. It evaluates the question, the table schema, and the outputs from both the NumSolver and the SOTA branch to select the final response.
3. TwEvaluator (Trustworthiness Evaluation)
This component introduces "regret-awareness." Using a Multi-Arm Bandit (MAB) algorithm, the TwEvaluator tracks the historical performance of different branches. It generates a trustworthiness label (True/False) alongside the answer, alerting the user if the model is operating in a high-uncertainty zone.
Experimental Results
TabLaP was evaluated on three major benchmarks: WikiTableQuestions (WTQ), FTQ, and TabFact. The results demonstrate a significant leap over previous state-of-the-art (SOTA) methods:
- Accuracy Boost: TabLaP improved overall accuracy by 5.7% on WTQ and 5.8% on FTQ compared to previous leaders like Mix-SC.
- Numerical Superiority: On questions specifically requiring arithmetic, TabLaP outperformed baselines by up to 6.8%.
- Reduced Regret: The trustworthiness labels successfully reduced the "user regret ratio" by approximately 20%, meaning users were significantly less likely to be misled by incorrect model outputs.
Conclusion
By decoupling reasoning from calculation and introducing a dynamic trustworthiness evaluator, TabLaP provides a robust solution to the numerical limitations of LLMs in structured data tasks. Future work will focus on expanding the framework to handle more complex relational databases and refining the rejection filters for even greater reliability.
Summary updated