Realistic Data Augmentation Framework for Enhancing Tabular Reasoning

Dibyakanti Kumar, Vivek Gupta, Soumya Sharma, Shuo Zhang

Abstract

Tabular Natural Language Inference (NLI) requires models to verify statements against structured data, a task where standard NLP models often struggle. We present a semi-automated framework designed to generate high-quality tabular reasoning data at scale. By utilizing human-authored templates and "rational counterfactual" tables, the framework produces the AUTO-TNLI dataset, containing over 1.4 million examples. Experimental results demonstrate that augmenting training with AUTO-TNLI significantly improves model robustness and performance, especially when human-labeled data is limited.

Introduction

Teaching machines to reason over structured data, such as Wikipedia infoboxes, is a fundamental challenge in Natural Language Processing. Tabular NLI involves determining if a hypothesis is supported (Entailment) or refuted (Contradiction) by a given table. Existing approaches to data collection face a difficult trade-off: human crowdsourcing is high-quality but expensive and prone to annotation bias, while fully automated generation often produces linguistically poor or logically simplistic examples. This research seeks a "Goldilocks" solution—a method as scalable as an algorithm but as logically sound as human reasoning.

Methodology: A Semi-Automated Framework

The proposed framework combines human expertise with automated generation to create diverse and logically consistent training data. The process consists of four primary stages:

  1. Hypothesis Template Creation: Humans design templates based on common table attributes (keys). For instance, a "Person" category template might be <Person_Name> died before <Died:Year>.
  2. Rational Counterfactual Table Creation: To prevent models from relying on spurious correlations or memorized facts, the system generates "fake" versions of real tables. These are "rational" in that they maintain internal logical consistency (e.g., ensuring a birth date remains earlier than a death date).
  3. Premise Paraphrasing: To enhance linguistic variety, templates are paraphrased (e.g., "is a graduate of" vs. "earned a degree from").
  4. Automatic Generation: The system populates these templates using thousands of real-world Wikipedia tables.
Overview of the semi-automated framework for generating tabular reasoning data, illustrating the flow from original tables to hypothesis generation.

The AUTO-TNLI Dataset

The framework was used to create AUTO-TNLI, a massive dataset comprising 1,478,662 pairs derived from over 10,000 tables across various categories, including "Person," "Movie," and "City." This scale is achieved by recognizing that tables within the same category share identical keys; a reasoning rule developed for one athlete's "Born" field can be applied to thousands of others.

A critical component of this dataset is the use of counterfactuals. By altering table data while keeping sentence structures constant, the model is forced to actively "read" the table rather than guessing based on linguistic patterns.

Example of original and counterfactual tables for Janet Leigh. The counterfactual table modifies specific values (e.g., death date, occupation) to generate new Entailment (E) and Contradiction (C) labels.

Experimental Results

The researchers evaluated the impact of AUTO-TNLI using RoBERTa and ALBERT models, yielding several key insights:

  • High Quality at Scale: Human evaluators confirmed the dataset's quality, with 99.4% correct labels and a high average grammar score of 4.89/5.
  • Performance in Low-Resource Settings: One of the most significant findings is the framework's effectiveness when human-labeled data is scarce. Adding AUTO-TNLI to a model trained on only 5% of the human-curated INFOTABS dataset allowed it to perform nearly as well as a model trained on the full human dataset.
  • Robustness Against Adversaries: Models trained with this framework showed a performance boost of 15-20% on adversarial datasets designed to trip up models that rely on simple pattern matching.

Conclusion

While the framework significantly advances tabular reasoning, it currently focuses on entity-centric "infobox" tables and does not yet address complex, multi-row statistical tables. Furthermore, the current iteration only generates Entailment and Contradiction labels, omitting the Neutral category often found in real-world NLI tasks. Future research may explore neural generation methods to further diversify templates and reduce human effort.

Summary updated

Abstract

Existing approaches to constructing training data for Natural Language Inference (NLI) tasks, such as for semi-structured table reasoning, are either via crowdsourcing or fully automatic methods. However, the former is expensive and time-consuming and thus limits scale, and the latter often produces naive examples that may lack complex reasoning. This paper develops a realistic semi-automated framework for data augmentation for tabular inference. Instead of manually generating a hypothesis for each table, our methodology generates hypothesis templates transferable to similar tables. In addition, our framework entails the creation of rational counterfactual tables based on human written logical constraints and premise paraphrasing. For our case study, we use the InfoTabs, which is an entity-centric tabular inference dataset. We observed that our framework could generate human-like tabular inference examples, which could benefit training data augmentation, especially in the scenario with limited supervision.
BibTeX
@inproceedings{kumar2022realistic,
  author = {Dibyakanti Kumar and Vivek Gupta and Soumya Sharma and Shuo Zhang},
  title = {{Realistic Data Augmentation Framework for Enhancing Tabular Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=5u3peuatgy1},
}
Figure