Rather a Nurse than a Physician -- Contrastive Explanations under Investigation

Oliver Eberle, Ilias Chalkidis, Laura Cabello, Stephanie Brandl

TL;DR

It is a common assumption in Explainable AI (XAI) that contrastive explanations ("Why X instead of Y?") are inherently closer to human reasoning than non-contrastive ones ("Why X?"). This paper empirically challenges this belief, showing that human-annotated rationales do not fundamentally differ between the two settings, and that standard machine explanations align equally well with both.

What Problem Does This Paper Address?

As machine learning models find deployment in high-stakes domains, the demand for transparent, human-plausible explanations has grown. A popular paradigm in XAI is contrastive explanation, which explains a model's choice relative to an alternative option (the "foil"). The intuition—backed by classic philosophical literature—is that humans naturally think contrastively.

While previous studies show that humans prefer receiving contrastive explanations, no one had empirically verified whether humans actually produce different rationales when asked to explain something contrastively versus non-contrastively. This paper investigates this missing link: Do human rationales actually change when shifting to a contrastive mindset, and do contrastive XAI methods align better with human logic?

Figure 1: An example from the BIOS dataset of non- contrastive and contrastive human and model-based rationales. Human rationales are underlined and bold- faced, while model-based rationale attribution scores are highlighted in red (positive) or blue (negative) colors.

Methods

The authors evaluate three post-hoc explainability methods across three distinct model architectures (RoBERTa, GPT-2, and T5) spanning three sizes (Small, Base, Large):

  1. Layer-wise Relevance Propagation (LRP): Adapted specifically to handle the non-linearities of Transformer architectures (like layer normalization and attention heads).
  2. Gradient × Input: A standard gradient-based feature attribution method.
  3. Gradient Norm: The direct norm of the gradients.

To formalize a contrastive explanation, the authors define the evidence as the difference between the logit score of the predicted target class ($L$) and the logit score of the foil class ($F$):

$$\text{Evidence} = y(x)_L - y(x)_F$$

To compare continuous model attribution scores with binary human highlights, the model scores are binarized by selecting the top-$k$ tokens, where $k$ matches the length of the human rationale.

Datasets

The study evaluates four English text-classification datasets:

  • SST2 & DynaSent: Binary and ternary sentiment analysis benchmarks used to analyze low-class-count dynamics.
  • Medical BIOS: A newly curated subset of the BIOS dataset focusing on five medical professions (psychologist, surgeon, nurse, dentist, physician). Crucially, the authors collected and released human rationale annotations for 100 samples across both contrastive and non-contrastive prompt settings.
  • DBPedia Animals: An 8-class dataset consisting of Wikipedia article abstracts for animal species.

Key Insights and Ideas

  • Binary Redundancy: In binary classification tasks (like SST2), contrastive and non-contrastive explanations are virtually identical (correlation $\approx 0.99$). This occurs because the model inherently treats the only alternative class as the foil during standard logit optimization.
  • Sparser, Not Different: In multi-class settings, when humans are asked "Why X instead of Y?", they do not choose entirely different words; instead, they choose fewer words. Human contrastive rationales are significantly more compact (averaging 4 words vs. 8 words in non-contrastive settings), acting as a precision filter rather than a strategic shift.
  • The Overlap Condition: Contrastive explanations only provide distinct utility when the target class and the foil class share a highly overlapping feature set (e.g., distinguishing a dentist from a surgeon). If the classes are semantically distant, the contrastive explanation defaults back to the non-contrastive baseline.
  • Pervasive Model Bias: While human rationales completely ignore gendered words, models heavily rely on pronouns (e.g., "he", "she") to predict occupations like nurse or surgeon, a bias clearly captured by the XAI methods.
Figure 6: Relative frequency of gendered words among the top-5 tokens in explanations.

Key Results

  • High Correlation: Model-based contrastive and non-contrastive explanations correlate strongly across the board, with Spearman coefficients reaching up to $0.99$ for RoBERTa models on DBpedia-Animals.
  • Stable Human Agreement: Inter-annotator agreement (measured via Cohen's Kappa) across contrastive and non-contrastive human annotations is highly stable (ranging from 0.40 to 0.73), mirroring the agreement levels found within the same setting.
  • Equivalent Alignment: As illustrated in Figure 5, model-generated explanations do not show a systematic advantage in aligning with human rationales when switching to a contrastive framework. Both explanation types perform comparably.
Figure 5: Agreement between human rationales and model-based explanations computed with LRP. Upper: RoBERTa, Center: GPT-2, Lower: T5
  • Lower Entropy: Human contrastive rationales exhibit significantly lower sentence-level entropy than non-contrastive ones ($1.14$ vs $1.72$, $p < 0.05$), quantitatively confirming that contrastive prompting forces humans to be more selective.

Limitations and Future Work

The authors note several boundaries to their current work:

  1. Task Complexity: The study is limited to English text classification with a small number of classes (up to 8). The benefits of contrastive explanations might only truly shine in complex tasks featuring hundreds of semantically similar labels.
  2. Binarization Loss: Converting continuous model attributions into binary masks to match human annotations discards nuanced gradient information.
  3. Post-Hoc Nature: Both human and model rationales were evaluated post-hoc (after a decision was made). Investigating pre-hoc human signals, such as eye-tracking gaze data during active reasoning, remains an open avenue for deeper alignment studies.

Summary updated

Abstract

Contrastive explanations, where one decision is explained in contrast to another, are supposed to be closer to how humans explain a decision than non-contrastive explanations, where the decision is not necessarily referenced to an alternative. This claim has never been empirically validated. We analyze four English text-classification datasets (SST2, DynaSent, BIOS and DBpedia-Animals). We fine-tune and extract explanations from three different models (RoBERTa, GTP-2, and T5), each in three different sizes and apply three post-hoc explainability methods (LRP, GradientxInput, GradNorm). We furthermore collect and release human rationale annotations for a subset of 100 samples from the BIOS dataset for contrastive and non-contrastive settings. A cross-comparison between model-based rationales and human annotations, both in contrastive and non-contrastive settings, yields a high agreement between the two settings for models as well as for humans. Moreover, model-based explanations computed in both settings align equally well with human rationales. Thus, we empirically find that humans do not necessarily explain in a contrastive manner.9 pages, long paper at ACL 2022 proceedings.
BibTeX
@inproceedings{eberle2023rather,
  author = {Oliver Eberle and Ilias Chalkidis and Laura Cabello and Stephanie Brandl},
  title = {{Rather a Nurse than a Physician -- Contrastive Explanations under Investigation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.427},
}
Figure