Being Right for Whose Right Reasons?

Terne Sasha Thorn Jakobsen, Laura Cabello, Anders Søgaard

Abstract

As machine learning models are increasingly deployed in sensitive domains, the demand for explainability has grown. A common metric for trustworthiness is whether a model is "right for the right reasons"—that is, whether its internal logic aligns with human rationales. However, this paper challenges the assumption that human rationales are an objective gold standard. By introducing a new dataset of demographic-augmented annotations, we demonstrate that NLP models systematically align better with the reasoning patterns of older and white individuals. Furthermore, we identify a "scaling law paradox" where larger models, despite higher predictive accuracy, exhibit lower alignment with human reasoning across all demographic groups.

Introduction

In the quest for trustworthy AI, researchers often evaluate model interpretability by comparing a model's internal logic to human-provided evidence. This "Right for the Right Reasons" (RRR) approach assumes that human reasoning is monolithic. This paper argues that what constitutes a "reason" is inherently subjective and influenced by a person's background. If a model’s explanations only align with the logic of a specific demographic (e.g., older, white users), the model may be less robust or even biased against other groups, even if its final predictions appear accurate. This study investigates whose "right reasons" current NLP models are actually learning and identifies systemic demographic biases in explainability methods.

Methodology

The researchers employed a multi-step experimental framework to probe the relationship between model rationales and human demographics:

  1. Demographic Annotation: 744 annotators were recruited via Prolific, balanced across age (Young < 36, Old > 37) and ethnicity (Black, Latino, White).
  2. Rationale Collection: Annotators provided both a label (e.g., "Positive" sentiment) and a rationale (highlighting specific words) for three NLP tasks.
  3. Model Probing: 16 different Transformer-based models, ranging from DistilBERT to DeBERTa-v3, were fine-tuned on these tasks.
  4. Explainability (XAI) Extraction: Two post-hoc methods were used to extract model rationales:
  5. Attention Rollout (AR): Accumulates attention weights across layers to approximate token importance.
  6. Layer-wise Relevance Propagation (LRP): A gradient-based method that backpropagates relevance from the output to the input tokens.
  7. Alignment Metrics: Token-F1 and Intersection-Over-Union (IOU-F1) were used to measure the overlap between model-highlighted words and human-highlighted words.

Datasets

The study augmented three existing NLP benchmarks with new demographic-specific rationale annotations:

  • DynaSent: A dynamic sentiment analysis dataset.
  • SST-2 (Stanford Sentiment Treebank): A standard movie review sentiment dataset.
  • CoS-E (Common Sense Explanations): A dataset for common-sense reasoning.

These augmented rationales have been made publicly available to facilitate further research into algorithmic fairness and interpretability.

Key Insights and Results

The Subjectivity of Rationales

The data reveals that even when different demographic groups agree on a final label, they often disagree on the evidence. For instance, Black Young (BY) annotators showed the lowest rationale alignment with other groups in sentiment tasks, while White Old (WO) annotators tended to provide the highest "consensus" rationales. This suggests that "ground truth" rationales are not neutral but reflect the perspectives of the majority or dominant demographic groups.

Systematic Demographic Bias

Models are not demographically neutral in their reasoning. Across almost all tested architectures, model rationales aligned significantly better with older and white annotators. In the SST-2 dataset, models aligned best with White Young (WY) and White Old (WO) groups. In DynaSent, there was a clear preference for "Old" groups across all ethnicities, indicating that models are learning to prioritize reasoning patterns that do not represent the full spectrum of human users.

The Scaling Law Paradox

One of the most surprising findings is the negative correlation between model size and rationale alignment. While larger models generally achieve higher predictive accuracy, Spearman correlation coefficients ($\rho$) between Token-F1 and model size were negative (e.g., $\rho = -0.4$ for DynaSent using Attention Rollout). As models grow in complexity, their internal "logic" moves further away from human-intuitive reasoning patterns, even as they become better at predicting the correct label.

Model Performance Outliers

DistilRoBERTa (dr) emerged as a consistent high-performer in rationale alignment across both sentiment datasets. Despite being a smaller, distilled model, it maintained higher alignment scores than many of its larger counterparts, though distillation alone did not close the gap between demographic groups.

Discussion and Limitations

While this study provides a critical look at demographic bias in XAI, it is limited to non-autoregressive Transformer models. Future research should investigate whether Large Language Models (LLMs) like GPT-4 exhibit similar biases. Additionally, the process of converting continuous XAI scores into binary rationales via a threshold is a simplification of model behavior.

To move toward more inclusive AI, the authors suggest that human rationales from diverse groups should be integrated directly into the training phase. Rationale-augmented learning could ensure that models learn to reason in a way that is representative of all users, rather than defaulting to the logic of a privileged few.

Summary updated

Abstract

Explainability methods are used to benchmark the extent to which model predictions align with human rationales i.e., are 'right for the right reasons'. Previous work has failed to acknowledge, however, that what counts as a rationale is sometimes subjective. This paper presents what we think is a first of its kind, a collection of human rationale annotations augmented with the annotators demographic information. We cover three datasets spanning sentiment analysis and common-sense reasoning, and six demographic groups (balanced across age and ethnicity). Such data enables us to ask both what demographics our predictions align with and whose reasoning patterns our models' rationales align with. We find systematic inter-group annotator disagreement and show how 16 Transformer-based models align better with rationales provided by certain demographic groups: We find that models are biased towards aligning best with older and/or white annotators. We zoom in on the effects of model size and model distillation, finding -- contrary to our expectations -- negative correlations between model size and rationale agreement as well as no evidence that either model size or model distillation improves fairness.
BibTeX
@inproceedings{jakobsen2023being,
  author = {Terne Sasha Thorn Jakobsen and Laura Cabello and Anders Søgaard},
  title = {{Being Right for Whose Right Reasons?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.59},
}
Figure