Being Right for Whose Right Reasons?

Terne Sasha Thorn Jakobsen, Laura Cabello, Anders Søgaard

TL;DR

Current AI explainability benchmarks assume there is a single "correct" reason for a prediction, but this paper proves that human reasoning is subjective and varies across demographics. The authors find that popular Transformer models are systematically biased, aligning their "rationales" significantly better with older and white annotators than with younger or minority groups.

What Problem Does This Paper Address?

In the quest for trustworthy AI, researchers use explainability (XAI) methods to ensure models are "right for the right reasons"—meaning they aren't just guessing based on statistical noise. Usually, this involves comparing a model's internal logic (rationales) against a "gold standard" provided by human annotators.

However, this approach has a major blind spot: it assumes all humans reason the same way. If a model aligns perfectly with a 60-year-old white man's reasoning but fails to match a 20-year-old Black woman's logic, is it truly "explainable" or just biased? This paper addresses the lack of demographic-aware rationale datasets and investigates whether model explanations are equally representative of all users.

Methods

The researchers developed a novel experimental setup to measure how well model logic aligns with different human groups, as illustrated in Figure 1.

Figure 1: Experimental setup for a sentiment analy- sis task. For a given instance, annotators are asked to choose a label and mark supporting evidence for their choice. For instances with full label agreement, we compare alignment of rationales (group-group align- ment). We do the same to measure group-model align- ment through attention- and gradient-based explainabil- ity methods.

Key components of their approach include:

  • Demographic Balancing: They recruited annotators via Prolific, balancing across six sub-groups based on age (Young <36, Old >37) and ethnicity (Black/African American, Latino/Hispanic, White/Caucasian).
  • Subjective Annotation: Unlike standard datasets where annotators justify a pre-given label, these annotators chose their own labels and then highlighted the words (extractive rationales) that supported their choice.
  • Model Probing: They evaluated 16 different Transformer-based models (ranging from tiny DistilBERT to large DeBERTa-v3-large).
  • XAI Techniques: They used two post-hoc explainability methods—Attention Rollout and Layer-wise Relevance Propagation (LRP)—to extract the models' internal rationales for comparison.

Datasets

The authors augmented three existing datasets with demographic-specific rationale annotations:

  1. DynaSent: A dynamic sentiment analysis dataset based on customer reviews.
  2. SST-2: The Stanford Sentiment Treebank (movie reviews).
  3. CoS-E: The Commonsense Explanations dataset, which requires reasoning beyond simple word associations.

The final dataset consists of 7,458 high-quality annotations, providing a unique lens into how different groups justify their linguistic decisions.

Key Insights and Ideas

  • Rationales are Subjective: There is systematic disagreement between demographic groups on why a sentence has a certain sentiment. For example, the "Black Young" (BY) group often showed the lowest alignment with other groups in sentiment tasks.
  • Models Have a "Preferred" Demographic: Models consistently align better with the rationales of older and white annotators. This suggests that the training data or the model architectures themselves are biased toward specific socio-demographic reasoning patterns.
  • The Model Size Paradox: While larger models generally achieve higher accuracy (label agreement), they actually show lower alignment with human rationales. As models get bigger, their internal "reasons" become less human-like.
  • Distillation Helps Plausibility, Not Fairness: Knowledge distillation (compressing large models into smaller ones) improves the overall human-likeness of rationales but does nothing to close the fairness gap between demographic groups.

Key Results

The study revealed a clear bias in how models "think." As shown in Figure 5, model rationales (extracted via LRP) align significantly better with older annotators (BO, LO, WO) across almost all ethnic groups in sentiment tasks.

Figure 5: Box-plots of group-model rationale alignment. Models align better with older annotators.

Quantitatively, the researchers found:

  • Negative Correlation with Size: Rationale agreement (Token-F1) was negatively correlated with model size (Spearman $\rho \approx -0.4$ for Attention Rollout). Larger models are "right" more often, but for increasingly "wrong" (or at least non-human) reasons.
  • Fairness Gaps: The "min-max gap"—the difference in alignment between the most-favored and least-favored demographic—did not improve with model size, as seen in Figure 6.
Figure 4: Group-group rationale agreement for instances with full label agreement. Agreement is measured by token-level binary F1. On the left side, average and std (error bar) token-F1 for 20 random combinations of paired group rationales over all datasets. On the right, each group-group agreement for each dataset. We observe lower agreement for BY except in CoS-E. WO tends to agree more with other groups, especially in CoS-E.

Limitations and Future Work

The authors acknowledge several limitations:

  • Model Scope: The study focused on non-autoregressive Transformers; results might differ for generative models like GPT-4.
  • Hyperparameters: Models were fine-tuned with standard settings rather than being optimized for each specific task, which might affect individual model performance.
  • XAI Binarization: Converting continuous importance scores from XAI methods into binary "rationales" (top-$k$ tokens) is a sensitive process that can influence agreement metrics.

Future Work: The researchers suggest that future fairness audits must look beyond simple accuracy and start examining the robustness of model logic across diverse populations. If a model is "right" for reasons that only make sense to one demographic, it may fail unexpectedly when deployed in a diverse real-world environment.

Summary updated

Abstract

Explainability methods are used to benchmark the extent to which model predictions align with human rationales i.e., are 'right for the right reasons'. Previous work has failed to acknowledge, however, that what counts as a rationale is sometimes subjective. This paper presents what we think is a first of its kind, a collection of human rationale annotations augmented with the annotators demographic information. We cover three datasets spanning sentiment analysis and common-sense reasoning, and six demographic groups (balanced across age and ethnicity). Such data enables us to ask both what demographics our predictions align with and whose reasoning patterns our models' rationales align with. We find systematic inter-group annotator disagreement and show how 16 Transformer-based models align better with rationales provided by certain demographic groups: We find that models are biased towards aligning best with older and/or white annotators. We zoom in on the effects of model size and model distillation, finding -- contrary to our expectations -- negative correlations between model size and rationale agreement as well as no evidence that either model size or model distillation improves fairness.
BibTeX
@inproceedings{jakobsen2023being,
  author = {Terne Sasha Thorn Jakobsen and Laura Cabello and Anders Søgaard},
  title = {{Being Right for Whose Right Reasons?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.59},
}
Figure