TL;DR
Current AI explainability benchmarks assume there is a single "correct" reason for a prediction, but this paper proves that human reasoning is subjective and varies across demographics. The authors find that popular Transformer models are systematically biased, aligning their "rationales" significantly better with older and white annotators than with younger or minority groups.
What Problem Does This Paper Address?
In the quest for trustworthy AI, researchers use explainability (XAI) methods to ensure models are "right for the right reasons"—meaning they aren't just guessing based on statistical noise. Usually, this involves comparing a model's internal logic (rationales) against a "gold standard" provided by human annotators.
However, this approach has a major blind spot: it assumes all humans reason the same way. If a model aligns perfectly with a 60-year-old white man's reasoning but fails to match a 20-year-old Black woman's logic, is it truly "explainable" or just biased? This paper addresses the lack of demographic-aware rationale datasets and investigates whether model explanations are equally representative of all users.
Methods
The researchers developed a novel experimental setup to measure how well model logic aligns with different human groups, as illustrated in Figure 1.
Key components of their approach include:
- Demographic Balancing: They recruited annotators via Prolific, balancing across six sub-groups based on age (Young <36, Old >37) and ethnicity (Black/African American, Latino/Hispanic, White/Caucasian).
- Subjective Annotation: Unlike standard datasets where annotators justify a pre-given label, these annotators chose their own labels and then highlighted the words (extractive rationales) that supported their choice.
- Model Probing: They evaluated 16 different Transformer-based models (ranging from tiny DistilBERT to large DeBERTa-v3-large).
- XAI Techniques: They used two post-hoc explainability methods—Attention Rollout and Layer-wise Relevance Propagation (LRP)—to extract the models' internal rationales for comparison.
Datasets
The authors augmented three existing datasets with demographic-specific rationale annotations:
- DynaSent: A dynamic sentiment analysis dataset based on customer reviews.
- SST-2: The Stanford Sentiment Treebank (movie reviews).
- CoS-E: The Commonsense Explanations dataset, which requires reasoning beyond simple word associations.
The final dataset consists of 7,458 high-quality annotations, providing a unique lens into how different groups justify their linguistic decisions.
Key Insights and Ideas
- Rationales are Subjective: There is systematic disagreement between demographic groups on why a sentence has a certain sentiment. For example, the "Black Young" (BY) group often showed the lowest alignment with other groups in sentiment tasks.
- Models Have a "Preferred" Demographic: Models consistently align better with the rationales of older and white annotators. This suggests that the training data or the model architectures themselves are biased toward specific socio-demographic reasoning patterns.
- The Model Size Paradox: While larger models generally achieve higher accuracy (label agreement), they actually show lower alignment with human rationales. As models get bigger, their internal "reasons" become less human-like.
- Distillation Helps Plausibility, Not Fairness: Knowledge distillation (compressing large models into smaller ones) improves the overall human-likeness of rationales but does nothing to close the fairness gap between demographic groups.
Key Results
The study revealed a clear bias in how models "think." As shown in Figure 5, model rationales (extracted via LRP) align significantly better with older annotators (BO, LO, WO) across almost all ethnic groups in sentiment tasks.
Quantitatively, the researchers found:
- Negative Correlation with Size: Rationale agreement (Token-F1) was negatively correlated with model size (Spearman $\rho \approx -0.4$ for Attention Rollout). Larger models are "right" more often, but for increasingly "wrong" (or at least non-human) reasons.
- Fairness Gaps: The "min-max gap"—the difference in alignment between the most-favored and least-favored demographic—did not improve with model size, as seen in Figure 6.
Limitations and Future Work
The authors acknowledge several limitations:
- Model Scope: The study focused on non-autoregressive Transformers; results might differ for generative models like GPT-4.
- Hyperparameters: Models were fine-tuned with standard settings rather than being optimized for each specific task, which might affect individual model performance.
- XAI Binarization: Converting continuous importance scores from XAI methods into binary "rationales" (top-$k$ tokens) is a sensitive process that can influence agreement metrics.
Future Work: The researchers suggest that future fairness audits must look beyond simple accuracy and start examining the robustness of model logic across diverse populations. If a model is "right" for reasons that only make sense to one demographic, it may fail unexpectedly when deployed in a diverse real-world environment.
Summary updated