Introduction
Modern NLP models often achieve high accuracy by exploiting "spurious correlations"—shortcuts in datasets that allow for correct predictions without true task understanding. For instance, in Natural Language Inference (NLI), a model might learn that the presence of the word "no" in a hypothesis strongly correlates with a "contradiction" label, regardless of the premise. To address this, researchers have introduced the Counterfactual Attentiveness Test (CAT), a method to quantify whether models are truly "listening" to all parts of their input.
The Counterfactual Attentiveness Test (CAT)
The CAT methodology is rooted in causal inference and designed to be fully automated. It tests model attentiveness by perturbing paired inputs (e.g., a Premise $P$ and a Hypothesis $H$).
The process follows four steps:
- Setup: Identify paired inputs $(x_1, x_2)$.
- Perturbation: Replace one part (e.g., $x_1$) with a random $x'_1$ from elsewhere in the dataset.
- Expectation: If the model is attentive to $x_1$, swapping it for an unrelated sentence should break the logical link, changing the prediction to a "default" label (e.g., Neutral).
- Metric: Attentiveness is the percentage of instances where the model's prediction changes following the swap.
Experimental Evaluation
The study evaluated 15 models across ten datasets and four major tasks:
- Natural Language Inference (NLI): MNLI, WANLI, RTE.
- Paraphrase Detection (PD): QQP, PAWS.
- Reading Comprehension (RC): SQuAD2.0, DuoRC, NewsQA.
- Visual & Language Reasoning (VLR): VQA2.0, NLVR2.
Key Findings: The ICL Paradox and Task Variance
The research reveals that attentiveness is highly data-dependent. While models are nearly 100% attentive on datasets like RTE, attentiveness on MNLI can drop as low as 46%.
A significant discovery is the In-Context Learning (ICL) Paradox. For large language models like GPT-3, increasing the number of demonstrations in a prompt improves standard accuracy but simultaneously decreases attentiveness. On MNLI, GPT-3's attentiveness fell from 92.9% (0-shot) to 62.0% (3-shot). This suggests that more examples may lead the model to rely on memorized label distributions rather than the specific input provided.
Furthermore, the study found that correlation does not always imply reliance. In Reading Comprehension, models remained highly attentive to context despite the existence of strong "question-only" shortcuts. In multimodal tasks like VQA, models showed a clear bias, being significantly more attentive to text (~89%) than to visual evidence (~65%).
Improving Model Focus
To mitigate these issues, the authors propose CAT Augmentation. By including random-swapped counterfactuals in the training or demonstration data, model attentiveness can be significantly boosted without compromising standard accuracy. Instruction tuning also appears to help, as seen with Flan-T5-base outperforming T5-base in attentiveness scores.
Limitations
CAT relies on the assumption that a random swap will result in a neutral label. While this holds for most datasets (92-100%), it failed on SNLI (~70%) due to its narrow lexicon. Future work aims to expand this framework to more complex input structures and broader instance types.
Summary updated