Abstract
Despite significant progress having been made in question answering on tabular data (Table QA), it's unclear whether, and to what extent existing Table QA models are robust to task-specific perturbations, e.g., replacing key question entities or shuffling table columns. To systematically study the robustness of Table QA models, we propose a benchmark called RobuT, which builds upon existing Table QA datasets (WTQ, WikiSQL-Weak, and SQA) and includes human-annotated adversarial perturbations in terms of table header, table content, and question. Our results indicate that both state-of-the-art Table QA models and large language models (e.g., GPT-3) with few-shot learning falter in these adversarial sets. We propose to address this problem by using large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models.
Introduction
Table Question Answering (Table QA) models have achieved impressive accuracy on standard benchmarks, demonstrating an ability to reason over structured data like spreadsheets and Wikipedia tables. However, these benchmarks often feature "clean" data where the natural language questions (NLQ) align closely with table headers and content. In real-world applications, data is rarely this pristine. Variations in terminology, structural changes like column shuffling, or paraphrased queries can cause these models to fail unexpectedly.
This research introduces RobuT, a comprehensive benchmark designed to stress-test Table QA models against human-annotated adversarial perturbations. By exposing the "shortcuts" models take to achieve high scores, the study provides a diagnostic tool for measuring true reasoning capabilities versus simple pattern matching.
The RobuT Benchmark
RobuT is constructed by applying perturbations to three foundational Table QA datasets: WTQ (WikiTableQuestions), WikiSQL-Weak, and SQA (Sequential Question Answering). The benchmark consists of 143,477 pairs of original and perturbed examples, categorized into three levels of adversarial modification:
- Table Header: Replacing column names with synonyms or related terms (e.g., changing "Runner-up" to "Second Place").
- Table Content: Modifying the structure of the table, such as shuffling rows and columns or decomposing a single column into multiple columns (e.g., splitting a "Score" column into "Home" and "Away").
- Natural Language Question (NLQ): Paraphrasing questions to remove linguistic cues that models often rely on (e.g., changing "How many..." to "What is the number of...").
To ensure the examples were truly adversarial, the researchers employed a model-in-the-loop approach. Human annotators were tasked with modifying tables or questions specifically until a target model (TaBERT) failed to produce the correct answer, ensuring the benchmark targets the specific weaknesses of current architectures.
Evaluating Model Fragility
The study evaluated several state-of-the-art models, including TAPAS and TAPEX, alongside Large Language Models (LLMs) like GPT-3. The results revealed a significant "performance cliff":
- Structural Sensitivity: Even simple changes like row or column shuffling—which do not change the underlying data or the correct answer—led to dramatic performance drops.
- The Shortcut Trap: Models frequently rely on specific keywords (e.g., "many" triggering a count operation). When these keywords are paraphrased, the models often fail to perform the required reasoning.
- Performance Impact: On the WTQ subset, the TAPAS model's accuracy dropped by as much as 12.5% when faced with mixed perturbations.
While LLMs like GPT-3 (specifically text-davinci-003) demonstrated higher inherent robustness in few-shot settings compared to specialized models, they still experienced performance gaps between original and perturbed data.
LETA: LLM-Enhanced Table QA Augmentation
To address the fragility identified in RobuT, the authors developed LETA, a framework that uses LLMs to generate high-quality adversarial training data at scale. Recognizing that human annotation for 140,000+ examples is prohibitively expensive, LETA leverages the reasoning capabilities of models like GPT-3 and CodeX.
Using chain-of-thought prompting, LETA guides the LLM to simulate human-like perturbations. For example, the LLM might be prompted to "Decompose this column" or "Paraphrase this question while maintaining its meaning." These synthetic adversarial examples are then used to fine-tune smaller, more efficient models like TAPEX.
Key Results and Findings
The implementation of LETA yielded significant improvements in model robustness:
- Improved R-ACC: Fine-tuning with LETA-generated data improved Robustness Accuracy (R-ACC) across nearly all perturbation categories.
- Question Robustness: For the TAPEX model, LETA improved accuracy on perturbed questions by over 3% compared to standard rule-based data augmentation.
- Persistent Challenges: Table content perturbations, specifically "Column Extension" (splitting data across columns), remain the most difficult challenge for all models, including those trained with LETA.
Conclusion
The RobuT benchmark highlights a critical gap between the benchmark performance and the real-world reliability of Table QA models. While specialized models are prone to overfitting on structural and linguistic shortcuts, the LETA framework demonstrates that LLMs can be used as "robust teachers" to distill reasoning capabilities into smaller models. Future work will look toward addressing cell-value perturbations and narrowing the gap between synthetic LLM-generated perturbations and the creative adversarial strategies employed by humans.
Summary updated