Leveraging Data Recasting to Enhance Tabular Reasoning

Aashna Jena, Vivek Gupta, Manish Shrivastava, Julian Martin Eisenschlos

Introduction

Natural Language Inference (NLI) is a core task in language understanding that requires determining the logical relationship—entailment, contradiction, or neutral—between a premise and a hypothesis. While NLI is well-studied for unstructured text, Tabular Natural Language Inference (TNLI) introduces significant complexity. In TNLI, the premise is a structured table, requiring models to perform symbolic reasoning, such as counting, ranking, and aggregation, to verify a statement.

Developing robust TNLI models is currently hindered by a "Goldilocks" problem in data collection. Human-annotated datasets offer linguistic creativity but are expensive to scale and often contain "annotation artifacts"—spurious correlations that allow models to bypass actual reasoning. Conversely, synthetic data is highly scalable but often lacks linguistic diversity, resulting in "robotic" patterns that do not generalize well to real-world language. This paper introduces Data Recasting, a semi-automatic framework designed to bridge this gap by repurposing existing human-annotated tabular data from other tasks into high-quality training data for TNLI.

The Data Recasting Framework

Data Recasting transforms existing datasets—originally designed for Table-to-Text generation, Table Question Answering (TQA), or Semantic Parsing (SPT)—into NLI formats. The framework employs several key perturbation techniques to generate diverse training pairs:

  • Hypothesis Perturbation: Starting with a true statement (e.g., "The team won 5 games"), the system generates entailments by substituting entities with other valid data from the same table. Contradictions are created by swapping values with data from different rows or using WordNet to identify antonyms (e.g., changing "most" to "least").
  • Table Perturbation (Counterfactuals): Instead of altering the text, the framework modifies the table itself. By swapping specific cell values, a previously true statement becomes false. This forces the model to ground its reasoning in the specific table provided rather than relying on pre-trained world knowledge.
  • Linguistic Diversity: To ensure the generated hypotheses remain natural, the authors utilize a T5 model fine-tuned on the PAWS dataset to paraphrase the results. This step ensures the data maintains the complexity of human-authored text while benefiting from automated scaling.

Dataset Construction

The researchers applied this framework to five major source datasets, resulting in hundreds of thousands of new TNLI instances:

  1. WikiTableQuestions & FeTaQA (TQA): Questions and answers were converted into declarative statements.
  2. WikiSQL & Squall (SPT): Logical SQL queries were used to identify precise cell dependencies, allowing for highly accurate perturbations.
  3. ToTTo (T2TG): Descriptive table summaries were transformed into NLI pairs, contributing significantly to the volume of data (e.g., ToTTo-TNLI alone provides 850,000 samples).

Key Insights and Results

The study reveals that tables are uniquely suited for recasting because their structured nature—clear boundaries and column headers—makes it easy to identify interchangeable entities. This structure allows for the creation of "minimal pair" tables where only one or two cells differ, effectively preventing models from learning spurious correlations.

Performance Benchmarks

  • Zero-Shot Transfer: A model pre-trained on the recasted QA-TNLI dataset achieved 71.1% accuracy on the popular TabFact benchmark without seeing any TabFact training examples. This performance surpassed the original supervised TabFact baseline.
  • Effective Augmentation: Integrating recasted data into training pipelines improved final performance on TabFact by 17% over the baseline and 1.1% over previous state-of-the-art synthetic augmentation methods.
  • Preserved Complexity: The researchers found that the complexity of the original task is preserved through recasting. For instance, data derived from Squall (which involves complex SQL logic) proved to be the most challenging for models, suggesting that recasting successfully captures high-level reasoning requirements.

Limitations and Future Directions

While effective, the Data Recasting framework has limitations. Data sourced from Table-to-Text tasks tends to be more descriptive and may lack the complex inferential leaps (like "ranked second") found in human-annotated NLI. Additionally, the framework currently performs best on "database-like" tables and may struggle with nested or chronological structures where row-swapping could break internal logic.

Future work will focus on expanding the framework to support multi-lingual TNLI, particularly for agglutinative languages that require more sophisticated morphological analysis, and exploring ways to reduce domain mismatch between Wikipedia-sourced data and other target domains.

Summary updated

Abstract

Creating challenging tabular inference data is essential for learning complex reasoning. Prior work has mostly relied on two data generation strategies. The first is human annotation, which yields linguistically diverse data but is difficult to scale. The second category for creation is synthetic generation, which is scalable and cost effective but lacks inventiveness. In this research, we present a framework for semi-automatically recasting existing tabular data to make use of the benefits of both approaches. We utilize our framework to build tabular NLI instances from five datasets that were initially intended for tasks like table2text creation, tabular Q/A, and semantic parsing. We demonstrate that recasted data could be used as evaluation benchmarks as well as augmentation data to enhance performance on tabular NLI tasks. Furthermore, we investigate the effectiveness of models trained on recasted data in the zero-shot scenario, and analyse trends in performance across different recasted datasets types.
BibTeX
@inproceedings{jena2022leveraging,
  author = {Aashna Jena and Vivek Gupta and Manish Shrivastava and Julian Martin Eisenschlos},
  title = {{Leveraging Data Recasting to Enhance Tabular Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.328},
}
Figure