Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals

Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Raghavi Chandu, Vivek Srikumar, Sameer Singh, Noah A. Smith

Introduction

Modern NLP models often achieve high accuracy by exploiting "spurious correlations"—shortcuts in datasets that allow for correct predictions without true task understanding. For instance, in Natural Language Inference (NLI), a model might learn that the presence of the word "no" in a hypothesis strongly correlates with a "contradiction" label, regardless of the premise. To address this, researchers have introduced the Counterfactual Attentiveness Test (CAT), a method to quantify whether models are truly "listening" to all parts of their input.

The Counterfactual Attentiveness Test (CAT)

The CAT methodology is rooted in causal inference and designed to be fully automated. It tests model attentiveness by perturbing paired inputs (e.g., a Premise $P$ and a Hypothesis $H$).

The Counterfactual Attentiveness Test (CAT) workflow.

The process follows four steps:

  1. Setup: Identify paired inputs $(x_1, x_2)$.
  2. Perturbation: Replace one part (e.g., $x_1$) with a random $x'_1$ from elsewhere in the dataset.
  3. Expectation: If the model is attentive to $x_1$, swapping it for an unrelated sentence should break the logical link, changing the prediction to a "default" label (e.g., Neutral).
  4. Metric: Attentiveness is the percentage of instances where the model's prediction changes following the swap.

Experimental Evaluation

The study evaluated 15 models across ten datasets and four major tasks:

  • Natural Language Inference (NLI): MNLI, WANLI, RTE.
  • Paraphrase Detection (PD): QQP, PAWS.
  • Reading Comprehension (RC): SQuAD2.0, DuoRC, NewsQA.
  • Visual & Language Reasoning (VLR): VQA2.0, NLVR2.

Key Findings: The ICL Paradox and Task Variance

The research reveals that attentiveness is highly data-dependent. While models are nearly 100% attentive on datasets like RTE, attentiveness on MNLI can drop as low as 46%.

Attentiveness vs. Partial Input Correlations across different NLP tasks.

A significant discovery is the In-Context Learning (ICL) Paradox. For large language models like GPT-3, increasing the number of demonstrations in a prompt improves standard accuracy but simultaneously decreases attentiveness. On MNLI, GPT-3's attentiveness fell from 92.9% (0-shot) to 62.0% (3-shot). This suggests that more examples may lead the model to rely on memorized label distributions rather than the specific input provided.

Furthermore, the study found that correlation does not always imply reliance. In Reading Comprehension, models remained highly attentive to context despite the existence of strong "question-only" shortcuts. In multimodal tasks like VQA, models showed a clear bias, being significantly more attentive to text (~89%) than to visual evidence (~65%).

Improving Model Focus

To mitigate these issues, the authors propose CAT Augmentation. By including random-swapped counterfactuals in the training or demonstration data, model attentiveness can be significantly boosted without compromising standard accuracy. Instruction tuning also appears to help, as seen with Flan-T5-base outperforming T5-base in attentiveness scores.

Limitations

CAT relies on the assumption that a random swap will result in a neutral label. While this holds for most datasets (92-100%), it failed on SNLI (~70%) due to its narrow lexicon. Future work aims to expand this framework to more complex input structures and broader instance types.

Summary updated

Abstract

The inevitable appearance of spurious correlations in training datasets hurts the generalization of NLP models on unseen data. Previous work has found that datasets with paired inputs are prone to correlations between a specific part of the input (e.g., the hypothesis in NLI) and the label; consequently, models trained only on those outperform chance. Are these correlations picked up by models trained on the full input data? To address this question, we propose a new evaluation method, Counterfactual Attentiveness Test (CAT). CAT uses counterfactuals by replacing part of the input with its counterpart from a different example (subject to some restrictions), expecting an attentive model to change its prediction. Using CAT, we systematically investigate established supervised and in-context learning models on ten datasets spanning four tasks: natural language inference, reading comprehension, paraphrase detection, and visual & language reasoning. CAT reveals that reliance on such correlations is mainly data-dependent. Surprisingly, we find that GPT3 becomes less attentive with an increased number of demonstrations, while its accuracy on the test data improves. Our results demonstrate that augmenting training or demonstration data with counterfactuals is effective in improving models’ attentiveness. We show that models’ attentiveness measured by CAT reveals different conclusions from solely measuring correlations in data.
BibTeX
@inproceedings{elazar2024measuring,
  author = {Yanai Elazar and Bhargavi Paranjape and Hao Peng and Sarah Wiegreffe and Khyathi Raghavi Chandu and Vivek Srikumar and Sameer Singh and Noah A. Smith},
  title = {{Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.205},
}
Figure