TL;DR
This paper introduces FREB-TQA, a novel benchmark designed to rigorously evaluate the robustness of Table Question Answering (TQA) systems across three critical dimensions: structural changes, attention to relevant cells, and numerical reasoning. Their extensive experiments reveal that current state-of-the-art TQA systems, including large language models, exhibit significant vulnerabilities across these aspects, highlighting the need for more robust TQA architectures.
What Problem Does This Paper Address?
Table Question Answering (TQA) systems aim to answer questions based on information presented in tables. While these systems have shown promise, prior research indicates a general lack of robustness. Understanding the specific reasons behind this fragility is crucial for developing more reliable TQA models. Existing benchmarks often provide a coarse-grained view of robustness, making it difficult to pinpoint whether failures stem from issues in retrieving relevant information, focusing on the correct data within a table, or performing accurate numerical reasoning. This paper addresses this gap by proposing a fine-grained evaluation framework to disentangle these aspects and identify specific weaknesses in TQA systems.
Methods
The core contribution of this paper is the FREB-TQA benchmark, which is built upon four existing TQA datasets (WTQ, WikiSQL, SQA, TAT). The authors introduce three key desiderata for evaluating TQA robustness:
- Retrieval Robustness against Table Structure Changes: This aspect tests if systems can retrieve correct cells regardless of table structure modifications. Perturbations include shuffling rows/columns, shifting target rows/columns to different positions, and transposing the entire table.
- Attention to Relevant Cells: This evaluates whether systems rely on relevant cell content or exploit shortcuts like model-internal knowledge or positional biases. Perturbations involve removing relevant cells or the entire table.
- Aggregation/Comparison Robustness against Value Changes: This assesses the system's ability to perform correct numerical aggregations and comparisons, even when cell values are altered. Perturbations involve modifying cell values to either change or not change the final answer.
The benchmark incorporates seven novel automatic perturbation methods and extensive human annotations to create a comprehensive evaluation suite.
Datasets
The FREB-TQA benchmark is constructed using development sets from four established TQA datasets:
- WTQ (Table Questions): Wikipedia-based, complex QA.
- WikiSQL: Wikipedia-based, simple QA.
- SQA (Sequential Question Answering): Wikipedia-based, conversational QA.
- TAT (Tabular And Textual dataset for Question Answering): Finance domain, complex QA.
The authors filtered out questions related to table structure itself.
Key Insights and Ideas
- Positional Bias is Pervasive: TQA systems exhibit a strong positional bias, meaning the location of the answer cell within a table significantly impacts performance. Systems tend to struggle more when the target row is moved further down the table.
- LLMs are Vulnerable to Structure, but Stronger in Numerics: While Large Language Models (LLMs) are more affected by row and column shuffling than end-to-end or pipeline systems, they demonstrate better performance on numerical reasoning tasks.
- Pipeline Systems Offer a Path to Robustness: Pipeline TQA models, which break down the problem into intermediate steps like query generation and execution, appear more robust to changes in relevant cells and values. This is attributed to their symbolic execution component.
- Fine-grained Evaluation is Crucial: The proposed benchmark effectively disentangles different sources of TQA system failure, providing deeper insights than coarse-grained robustness evaluations.
Key Results
- No System is Consistently Robust: Across all three robustness aspects, no single TQA system consistently performed well. System performances generally dropped significantly under various perturbations.
- Column Shuffling is More Damaging than Row Shuffling: Systems, particularly LLMs, are more vulnerable to changes in column order than row order.
- Transposing Tables is a Major Challenge: End-to-end TQA systems, in particular, struggle significantly with transposed tables, indicating a need for architectural improvements or pre-training on more diverse table structures.
- Pipeline Models Excel in Value Changes: Pipeline systems like TaPas and Binder showed greater robustness when cell values were modified, likely due to their direct use of cell values in execution.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Focus on Numerical Reasoning: The current benchmark primarily focuses on numerical reasoning. Future work could extend this to include commonsense or temporal reasoning.
- English-Centric: The benchmark is built on English TQA datasets. Expanding to other languages would be beneficial.
- Distinguishing Value Types: The benchmark does not explicitly distinguish between the impact of changes in numeric versus non-numeric values. Future work could explore this.
- Pipeline Systems for LLMs: The authors advocate for more research into pipeline-based TQA systems, suggesting the potential for integrating LLMs into a pipeline manner to build more robust systems.
- Handling Long Tables: Addressing the shortcomings of current systems in processing long tables remains an important area for future research.
Summary updated