Introduction
Modern Natural Language Processing (NLP) models often achieve high accuracy by "cheating"—relying on spurious correlations or statistical shortcuts in datasets rather than developing a deep understanding of the input. For instance, in Natural Language Inference (NLI) tasks, a model might learn that the presence of the word "never" in a hypothesis almost always correlates with a "Contradiction" label, regardless of what the premise says.
While previous research identified these biases through partial-input baselines, it remained unclear whether full-input models were actually exploiting these shortcuts. This paper introduces the Counterfactual Attentiveness Test (CAT) to quantify a model's attentiveness to its entire input and proposes a method to mitigate these biases.
The Counterfactual Attentiveness Test (CAT)
The CAT methodology is a "black-box" evaluation that does not require access to a model's internal weights. It operates on the principle of perturbation:
- Original Input: A model is given a paired input, such as a Premise ($x_1$) and a Hypothesis ($x_2$).
- Counterfactual Swap: The premise ($x_1$) is replaced with a random premise ($x'_1$) drawn from elsewhere in the dataset.
- Evaluation: If the model is truly attentive to the relationship between both inputs, its prediction should change to a "Neutral" or "No Answer" label, as the new random pair is highly unlikely to have a logical entailment or contradiction.
Attentiveness is measured as the percentage of instances where the model's prediction changes appropriately when the input is perturbed. A low CAT score indicates that the model is likely ignoring the premise and relying solely on the hypothesis.
Key Insights: Data Bias vs. Model Behavior
A critical finding of the study is that dataset bias does not always dictate model behavior. The researchers evaluated ten datasets across four tasks: Natural Language Inference (NLI), Paraphrase Detection (PD), Reading Comprehension (RC), and Visual & Language Reasoning (VLR).
As shown in the figure above, tasks like Reading Comprehension (QA) maintain high attentiveness despite having high spurious correlations in the data. Conversely, in NLI tasks, models are much more prone to exploiting shortcuts. This suggests that attentiveness is an emergent property of the interaction between a specific model architecture and the data it processes.
The GPT-3 Paradox
The study reveals a troubling trend in Large Language Models (LLMs) like GPT-3. During "In-Context Learning," as more training demonstrations are added to the prompt, GPT-3's accuracy improves, but its attentiveness decreases. Specifically, on the MNLI dataset, attentiveness dropped from 92.9% (zero-shot) to 62.0% (three-shot). This suggests that as models scale and receive more context, they may become more adept at identifying and exploiting superficial patterns rather than performing robust reasoning.
Improving Attentiveness with CAT Augmentation
To combat this lack of attentiveness, the authors propose CAT Augmentation. This involves adding counterfactual "unrelated" pairs (created via random swaps) into the training set. This forces the model to learn that it cannot rely on one part of the input alone; it must verify the relationship between both parts to make a valid prediction.
Results showed that CAT Augmentation is highly effective. For example, it raised the attentiveness of a T5-large model on MNLI from 71.5% to 99.7% without sacrificing standard accuracy.
Limitations and Conclusion
While CAT provides a powerful diagnostic tool, it is not a standalone metric for model quality; a model could be perfectly attentive but entirely inaccurate. Additionally, the test assumes that random swaps result in unrelated pairs, an assumption that can fail in datasets where inputs are highly homogeneous (such as SNLI).
Ultimately, CAT offers a necessary lens for evaluating model reliability, proving that even our most advanced models require targeted interventions to ensure they are reasoning over the full scope of the information provided.
Summary updated