Abstract
Given the increasingly prominent role NLP models (will) play in our lives, it is important for human expectations of model behavior to align with actual model behavior. Using Natural Language Inference (NLI) as a case study, we investigate the extent to which human-generated explanations of models' inference decisions align with how models actually make these decisions. More specifically, we define three alignment metrics that quantify how well natural language explanations align with model sensitivity to input words, as measured by integrated gradients. Then, we evaluate eight different models (the base and large versions of BERT, RoBERTa and ELECTRA, as well as an RNN and bag-of-words model), and find that the BERT-base model has the highest alignment with human-generated explanations, for all alignment metrics. Focusing in on transformers, we find that the base versions tend to have higher alignment with human-generated explanations than their larger counterparts, suggesting that increasing the number of model parameters leads, in some cases, to worse alignment with human explanations. Finally, we find that a model's alignment with human explanations is not predicted by the model's accuracy, suggesting that accuracy and alignment are complementary ways to evaluate models.
Introduction
As NLP models become more integrated into society, we don't just need them to be right; we need them to be right for the right reasons. Often, a model might achieve high accuracy by exploiting "spurious correlations" or shallow heuristics that a human would never use. This paper introduces Importance Alignment, a metric that quantifies how well a model's internal decision-making logic matches human expectations. By comparing "Integrated Gradients" (what the model actually looks at) with human-generated explanations, the authors explore the gap between human mental models of AI and the actual mathematical behavior of those systems.
Methods
The authors propose a framework to measure Importance Alignment using two main components:
- Model Importance: They use Integrated Gradients (IG), an attribution method that assigns an "importance score" to each input word based on how much it influenced the model's final prediction.
- Oracle Importance (The Human Side): To turn natural language explanations into word-level importance scores, the authors developed three "Oracles":
- Hard Oracle: A simple binary check—if a word from the input appears in the human explanation, it’s "important."
- Soft Oracle: A separate "explanation-informed" model is trained to predict the label using both the input and the human explanation, then IG is applied to that model.
- Expert Oracle: A "gold standard" where experts manually highlighted which input words they believed the human explainer was referring to.
The final Importance Alignment score is the correlation (Pearson’s $r$) between the Model Importance and the Oracle Importance.
Key Results
The study evaluated eight models on Adversarial NLI (ANLI), a dataset containing examples where humans successfully fooled models and then explained why.
The "Scaling Tax" on Interpretability
A surprising trend emerged: in the Transformer family (BERT, RoBERTa, ELECTRA), the "Base" versions consistently showed higher alignment with humans than the "Large" versions. This suggests that as we add more parameters to improve accuracy, models may find increasingly "alien" or non-human ways to process information.
Accuracy vs. Alignment
As shown in the results table below, BERT-Base achieved the highest importance alignment across metrics, despite being outperformed in raw accuracy by larger models like ELECTRA-Large. Furthermore, a model’s performance on standard test sets does not predict how well it aligns with human explanations. This indicates that alignment is a distinct dimension of model quality not captured by current accuracy-based benchmarks.
Discussion and Limitations
While Transformers aligned relatively well, simpler models like LSTMs (BInferSent) and Bag-of-Words (BOW) models had significantly lower alignment, suggesting that architectural complexity helps capture human-like reasoning only up to a point.
However, the study is limited to Natural Language Inference. It remains to be seen if these alignment trends hold for other tasks like Question Answering. Additionally, the paper identifies that larger models align worse, but the underlying cause—whether it is the pre-training objective, parameter count, or data volume—remains an open question for future research.
Summary updated