Abstract
Existing approaches to constructing training data for Natural Language Inference (NLI) tasks, such as for semi-structured table reasoning, are either via crowdsourcing or fully automatic methods. However, the former is expensive and time-consuming and thus limits scale, and the latter often produces naive examples that may lack complex reasoning. This paper develops a realistic semi-automated framework for data augmentation for tabular inference. Instead of manually generating a hypothesis for each table, our methodology generates hypothesis templates transferable to similar tables. In addition, our framework entails the creation of rational counterfactual tables based on human-written logical constraints and premise paraphrasing. For our case study, we use InfoTabs, an entity-centric tabular inference dataset. We observed that our framework could generate human-like tabular inference examples, which could benefit training data augmentation, especially in the scenario with limited supervision.
Introduction
Natural Language Inference (NLI) is a fundamental task in language understanding, requiring a model to determine if a hypothesis is an entailment, contradiction, or neutral relative to a given premise. While NLI for unstructured text is well-established, Tabular NLI—where the premise is a semi-structured table—presents unique challenges. Models must reason across rows and columns, often requiring a synthesis of multiple data points.
Currently, researchers face a "Goldilocks" problem in data collection. Crowdsourcing yields high-quality, complex reasoning but is prohibitively expensive and difficult to scale. Conversely, fully automatic generation scales effortlessly but often results in "naive" examples that lack logical depth or suffer from factual hallucinations. This paper introduces a semi-automatic framework designed to achieve the quality of human reasoning at the scale of machine generation.
The AUTO-TNLI Framework
The proposed framework utilizes a semi-automated pipeline to generate the AUTO-TNLI dataset. This approach relies on four primary pillars to ensure both scale and logical rigor:
1. Hypothesis Template Creation
Rather than writing individual sentences for every table, human annotators develop "rules" or templates based on table categories. For instance, tables in the "Person" category typically share keys like Born and Died. A template such as <Person_Name> died after <Died:Year> can be automatically applied to thousands of relevant tables. This method leverages the structural consistency of entity-centric tables to maximize human effort.
2. Rational Counterfactual Table Creation
A common failure mode in NLI models is the reliance on "spurious correlations" or internal knowledge from pre-training rather than the provided premise. To combat this, the framework generates "rational counterfactual" tables. By altering specific values while maintaining logical consistency (e.g., ensuring a death date remains chronologically after a birth date), the system forces the model to inspect the table values directly to determine the truth of a hypothesis.
3. Premise Paraphrasing
To improve linguistic diversity and prevent the model from over-fitting to specific table headers, the framework employs both manual and automated paraphrasing. This ensures that the model recognizes that "graduated from" and "is an alumnus of" refer to the same underlying relationship within the table.
4. Automated Generation and Scale
By populating these human-designed templates with data from 10,182 Wikipedia infoboxes, the framework generated 1,478,662 table-hypothesis pairs. This dataset covers 660 templates across 134 unique table keys, focusing on binary classification (Entailment vs. Contradiction).
Experimental Results
The authors evaluated the utility of AUTO-TNLI using RoBERTa and ALBERT models, focusing on its impact as a data augmentation tool for the human-curated INFOTABS benchmark.
Quality and Difficulty
Despite being semi-automatically generated, AUTO-TNLI is not trivial for existing models. Models trained on standard textual NLI (like MNLI) achieved only ~67.5% accuracy, suggesting the dataset requires genuine tabular reasoning. Furthermore, the template-based approach reached a human-verified accuracy of 99.4%, far surpassing the ~32.7% accuracy observed in fully automated LLM-based generation (e.g., GPT-J).
Performance Gains in Low-Supervision
The framework proved most effective in scenarios where human-annotated data was scarce. When training on only 5% of the original INFOTABS training set, augmenting with AUTO-TNLI provided a massive performance boost, increasing accuracy from approximately 50% to 62% on specific test splits.
Robustness
Models trained with the inclusion of rational counterfactuals demonstrated superior performance on adversarial test sets. By training on data where hypotheses were designed to trick the model, the resulting systems became more robust and less likely to rely on statistical shortcuts.
Conclusion and Future Work
The AUTO-TNLI framework demonstrates that semi-automated, template-based generation is a highly effective strategy for scaling tabular reasoning data without sacrificing logical integrity. While currently focused on entity-centric infoboxes and binary labels, future work aims to expand the framework to include "Neutral" labels and more complex, hierarchical table structures. This methodology provides a scalable blueprint for enhancing the reasoning capabilities of language models in structured data domains.
Summary updated