Abstract
As machine learning models are increasingly deployed in sensitive domains, the demand for explainability has grown. A common metric for trustworthiness is whether a model is "right for the right reasons"—that is, whether its internal logic aligns with human rationales. However, this paper challenges the assumption that human rationales are an objective gold standard. By introducing a new dataset of demographic-augmented annotations, we demonstrate that NLP models systematically align better with the reasoning patterns of older and white individuals. Furthermore, we identify a "scaling law paradox" where larger models, despite higher predictive accuracy, exhibit lower alignment with human reasoning across all demographic groups.
Introduction
In the quest for trustworthy AI, researchers often evaluate model interpretability by comparing a model's internal logic to human-provided evidence. This "Right for the Right Reasons" (RRR) approach assumes that human reasoning is monolithic. This paper argues that what constitutes a "reason" is inherently subjective and influenced by a person's background. If a model’s explanations only align with the logic of a specific demographic (e.g., older, white users), the model may be less robust or even biased against other groups, even if its final predictions appear accurate. This study investigates whose "right reasons" current NLP models are actually learning and identifies systemic demographic biases in explainability methods.
Methodology
The researchers employed a multi-step experimental framework to probe the relationship between model rationales and human demographics:
- Demographic Annotation: 744 annotators were recruited via Prolific, balanced across age (Young < 36, Old > 37) and ethnicity (Black, Latino, White).
- Rationale Collection: Annotators provided both a label (e.g., "Positive" sentiment) and a rationale (highlighting specific words) for three NLP tasks.
- Model Probing: 16 different Transformer-based models, ranging from DistilBERT to DeBERTa-v3, were fine-tuned on these tasks.
- Explainability (XAI) Extraction: Two post-hoc methods were used to extract model rationales:
- Attention Rollout (AR): Accumulates attention weights across layers to approximate token importance.
- Layer-wise Relevance Propagation (LRP): A gradient-based method that backpropagates relevance from the output to the input tokens.
- Alignment Metrics: Token-F1 and Intersection-Over-Union (IOU-F1) were used to measure the overlap between model-highlighted words and human-highlighted words.
Datasets
The study augmented three existing NLP benchmarks with new demographic-specific rationale annotations:
- DynaSent: A dynamic sentiment analysis dataset.
- SST-2 (Stanford Sentiment Treebank): A standard movie review sentiment dataset.
- CoS-E (Common Sense Explanations): A dataset for common-sense reasoning.
These augmented rationales have been made publicly available to facilitate further research into algorithmic fairness and interpretability.
Key Insights and Results
The Subjectivity of Rationales
The data reveals that even when different demographic groups agree on a final label, they often disagree on the evidence. For instance, Black Young (BY) annotators showed the lowest rationale alignment with other groups in sentiment tasks, while White Old (WO) annotators tended to provide the highest "consensus" rationales. This suggests that "ground truth" rationales are not neutral but reflect the perspectives of the majority or dominant demographic groups.
Systematic Demographic Bias
Models are not demographically neutral in their reasoning. Across almost all tested architectures, model rationales aligned significantly better with older and white annotators. In the SST-2 dataset, models aligned best with White Young (WY) and White Old (WO) groups. In DynaSent, there was a clear preference for "Old" groups across all ethnicities, indicating that models are learning to prioritize reasoning patterns that do not represent the full spectrum of human users.
The Scaling Law Paradox
One of the most surprising findings is the negative correlation between model size and rationale alignment. While larger models generally achieve higher predictive accuracy, Spearman correlation coefficients ($\rho$) between Token-F1 and model size were negative (e.g., $\rho = -0.4$ for DynaSent using Attention Rollout). As models grow in complexity, their internal "logic" moves further away from human-intuitive reasoning patterns, even as they become better at predicting the correct label.
Model Performance Outliers
DistilRoBERTa (dr) emerged as a consistent high-performer in rationale alignment across both sentiment datasets. Despite being a smaller, distilled model, it maintained higher alignment scores than many of its larger counterparts, though distillation alone did not close the gap between demographic groups.
Discussion and Limitations
While this study provides a critical look at demographic bias in XAI, it is limited to non-autoregressive Transformer models. Future research should investigate whether Large Language Models (LLMs) like GPT-4 exhibit similar biases. Additionally, the process of converting continuous XAI scores into binary rationales via a threshold is a simplification of model behavior.
To move toward more inclusive AI, the authors suggest that human rationales from diverse groups should be integrated directly into the training phase. Rationale-augmented learning could ensure that models learn to reason in a way that is representative of all users, rather than defaulting to the logic of a privileged few.
Summary updated