Abstract
To truly grasp reasoning ability, a Natural Language Inference (NLI) model should be evaluated on counterfactual data. TabPert facilitates this by assisting in the generation of such counterfactual data for assessing model tabular reasoning issues. TabPert allows a user to update a table, change its associated hypotheses, change their labels, and highlight rows that are important for hypothesis classification. TabPert also captures information about the techniques used to automatically produce the table, as well as the strategies employed to generate the challenging hypotheses. These counterfactual tables and hypotheses, as well as the metadata, can then be used to explore an existing model's shortcomings methodically and quantitatively.
Introduction
While large language models have demonstrated mastery over many text-based reasoning tasks, their performance on semi-structured data, such as tables, remains a significant challenge. Researchers suspect that these models often rely on "shortcuts"—spurious patterns or biases in the hypotheses—rather than performing genuine reasoning over the tabular evidence. To investigate this, counterfactual data is required: modified versions of tables and hypotheses that test if a model's prediction changes correctly when the underlying facts change. However, creating such data manually is slow and prone to logical inconsistencies. TabPert addresses these challenges by providing a streamlined, semi-automated platform for generating high-quality counterfactual tabular datasets.
The TabPert Platform
TabPert is a specialized web-based platform designed to facilitate adversarial tabular data creation while maintaining logical consistency. The interface is divided into several functional areas:
- Table Perturbation: Users can modify table values by dragging and dropping values from other tables in the same category or by manual entry. The tool includes type validation to prevent logical errors, such as placing a name in a date field.
- Hypothesis Perturbation: Annotators can modify hypotheses and their corresponding labels (Entailment, Neutral, Contradiction) side-by-side with the table.
- Strategy Selection and Metadata: For every change, the tool records the strategy used (e.g., flipping a label by changing the table vs. changing the hypothesis). This is captured in a 7-bit metadata string, allowing for granular analysis of model failures.
Experimental Setup
The researchers utilized a subset of the INFOTABS dataset, which contains Wikipedia-style entity tables. Using TabPert, they developed a new benchmark consisting of 109 counterfactual tables and 982 table-hypothesis pairs. This dataset was specifically engineered to mitigate "hypothesis bias," where a model predicts a label without considering the table content.
Results and Analysis
Evaluating a RoBERTa-Large model on this new data revealed significant performance gaps. While the model achieved 78.91% accuracy on the original INFOTABS data, its performance dropped to 61.26% on the counterfactual set. In contrast, human accuracy remained stable at approximately 85%, confirming that the counterfactuals are logically sound but difficult for AI.
Key findings include:
- Hypothesis Artifacts: When forced to predict labels without the table, model accuracy dropped from 64.32% to 44.85%, indicating that TabPert successfully removed superficial cues.
- Strategy Effectiveness: Manually changing the table to flip a label (Strategy 1) proved to be the most effective way to break the model, resulting in a 41.75% relative accuracy drop. This suggests models struggle to track how specific cell changes impact the truth of a statement.
- Efficiency: TabPert significantly improved annotation speed, reducing the time to process a table from over 40 minutes (using traditional spreadsheets) to just 7 minutes.
Conclusion
TabPert provides a robust framework for identifying and quantifying the reasoning flaws in NLI models. By exposing the reliance on superficial patterns through systematic counterfactual generation, the platform paves the way for developing models with more reliable tabular reasoning capabilities. Future work will focus on deploying the tool for collaborative use and further automating common-sense constraints.
Summary updated