Introduction
While Table Question Answering (Table QA) models like TAPAS and OmniTab have achieved impressive results on standard benchmarks, there is growing concern that these models rely on brittle shortcuts rather than deep reasoning. For instance, a model might fail if the table columns are reordered or if a header is replaced with a synonym. Existing robustness studies often rely on simple, rule-based perturbations that lack the linguistic complexity of real-world human interaction.
To address this, the authors introduce ROBUT, a massive benchmark of 138,149 human-annotated adversarial examples designed to systematically test the limits of Table QA systems.
The ROBUT Benchmark
ROBUT is built by perturbing the development sets of three foundational datasets: WikiTableQuestions (WTQ), WIKISQL-WEAK, and SQA. The benchmark categorizes perturbations into three levels:
- Table Header: Replacing column names with synonyms or abbreviations (e.g., "Runner-up" to "Second Place").
- Table Content: Shuffling rows/columns, adding irrelevant "distractor" columns, or masking columns that can be inferred from others.
- Natural Language Question (NLQ): Using a "model-in-the-loop" approach where humans rewrite questions to fool a specific model (TaBERT) while maintaining the original meaning.
This comprehensive suite allows for a diagnostic evaluation of where models fail, revealing whether they overfit to specific table layouts or linguistic patterns.
The LETA Framework
To mitigate the identified weaknesses, the authors propose LETA (LLM-Enhanced Table QA Augmentation). LETA leverages Large Language Models (LLMs) like GPT-3 and CodeX with Chain-of-Thought (CoT) prompting to generate high-quality adversarial training data. This approach mimics human-annotated perturbations but at a fraction of the cost—approximately $1.50 per 100 examples compared to $60.00 for human labor. This allows for the generation of massive amounts of diverse training data that mimics the "hard" examples found in the ROBUT benchmark.
Key Results and Insights
The evaluation of state-of-the-art (SOTA) models on ROBUT reveals a significant "performance collapse." Specialized Transformer-based models like TAPEX and OmniTab saw accuracy drops of up to 12.5% when faced with mixed perturbations.
LLM Resilience
Interestingly, while GPT-3's raw accuracy on original data was sometimes lower than fine-tuned specialized models, it exhibited much higher Robustness Accuracy (R-Acc). In many categories, GPT-3 maintained over 90% accuracy after perturbation, suggesting that its massive pre-training on diverse text provides a more stable foundation for understanding tabular data than task-specific fine-tuning alone.
Effectiveness of LETA
Fine-tuning models with LETA-generated synthetic data proved highly effective. For example, LETA improved TAPEX's accuracy on "Abbreviation Replacement" by 5.7%, outperforming standard rule-based augmentation techniques. This demonstrates that LLMs can serve as powerful engines for generating the "hard" examples needed to train more robust models.
Conclusion
The ROBUT benchmark exposes deep vulnerabilities in current Table QA systems, highlighting a tendency to rely on layout-based shortcuts. While LLMs show promising inherent robustness, the LETA framework provides a scalable path forward for improving specialized models. Future work will look into more complex perturbations, such as modifying cell values, and extending this robustness framework to tasks like table-to-text generation.
Summary updated