TL;DR
This paper introduces a novel approach to Table Question Answering (Table QA) that leverages Large Language Models (LLMs) to generate and execute code for extracting answers from tabular data. This "code-then-execute" strategy significantly improves accuracy and interpretability, while also reducing the number of LLM calls needed.
What Problem Does This Paper Address?
Answering questions over tabular data (Table QA) is a critical but challenging task in Natural Language Processing (NLP). Tables, with their diverse structures, nested headers, and implicit relationships, make it difficult for models to reason and retrieve precise information. While LLMs have shown promise, they still struggle with complex tables and queries, often producing answers that are hard to interpret or verify. The DataBench benchmark was introduced to address this, but achieving high performance remains a hurdle. This paper aims to bridge this gap by developing a system that can reliably and interpretably answer questions over tabular data.
Methods
The core of the proposed system is its "code-then-execute" methodology. It works by:
- Table Representation: Tables are uniformly represented as dataframes.
- Code Generation: An LLM is prompted with a natural language question and a preview of the table (first five rows) to generate executable code (likely Python using pandas) that can extract the answer.
- Verification and Execution: Two verification mechanisms are employed:
- Immediate Validation (Contemplator): The LLM is asked to assess the generated code's feasibility and correctness before execution.
- Post-Execution Debugging (Debugger): If the code fails to execute, the error message is fed back to the LLM, prompting it to regenerate a corrected version.
- Answer Extraction: The validated and corrected code is then executed to retrieve the final answer.
This approach aims to enhance interpretability by making the reasoning process explicit through code and improve efficiency by offloading complex computations to code execution.
Datasets
The primary dataset used for evaluation is DataBench. This benchmark consists of structured tables paired with natural language questions and their answers, covering diverse domains like finance, healthcare, and sports. DataBench includes complex queries requiring aggregation, filtering, and multi-hop reasoning. The dataset was provided in training, development, and test sets. The test set had two versions: Task Test_All (1468 rows) and Task Test_Lite (18 rows), with the latter significantly impacting answer accuracy due to its reduced size.
Key Insights and Ideas
- Code as an Intermediate Representation is Powerful: Generating executable code for table QA is a highly effective strategy, leading to improved accuracy and interpretability.
- The "Debugger" Approach Outperforms "Contemplator": Deferring code verification until an error occurs (the Debugger approach) leads to better performance than having the LLM self-assess code feasibility before execution (the Contemplator approach). This suggests that explicit error feedback is more beneficial for LLM correction.
- Explicit Answer Formatting Boosts Accuracy: Providing the LLM with an explicit desired output format for answers generally improves accuracy.
- Model Choice Matters Significantly: Transitioning from GPT-4o-mini to
ol-miniresulted in substantial accuracy improvements on test sets, though it came with increased computational cost. - Numerical Answers Pose a Challenge: Numerical answer accuracy is lower, potentially due to LLMs returning full-precision floats when answers are expected to be rounded to two decimal places.
- Table Size Impacts Performance: Smaller table sizes (like in
Test Lite) significantly degrade answer accuracy, especially for numerical responses.
Key Results
- The proposed "code-then-execute" approach achieved high accuracy on the DataBench benchmarks.
- The system, particularly when using
ol-mini, demonstrated significant performance improvements over state-of-the-art models reported in the original DataBench paper, with average scores improving by 13% to 28%. - GPT-4o-mini achieved the highest observed accuracy of 95.3% on Boolean questions within the validation set.
- The Debugger mode (verifying code after an error) significantly improved answer accuracy compared to other methods.
- While
ol-miniachieved the best test set accuracy, it incurred higher computational costs.
Limitations and Future Work
- LLM Self-Reflection Limitations: The system's capacity for self-reflection and self-error identification is limited, which has impeded further model improvement.
- API Instability: OpenAI's API can sometimes result in no code being generated ("None/Error").
- Numerical Precision: LLMs may return full-precision floats for numerical answers when rounding is expected.
- Ambiguity in Natural Language: Complex or lengthy sentences in queries can lead to LLMs focusing on initial instructions and overlooking crucial conditions.
- Model Hallucinations: Certain discrepancies in answers can be attributed to model hallucinations.
Future work could involve developing more robust self-reflection mechanisms for LLMs and exploring strategies to mitigate issues with API instability and numerical precision. The paper also hints at the potential of a "rewriter function" to clarify complex questions, though its overall impact was mixed.
Summary updated