TL;DR
Current NLP interpretability methods focus heavily on "faithfulness"—ensuring an explanation accurately reflects a model's inner workings. However, this paper reveals that technically faithful explanations can still be deeply misleading if they don't align with how humans naturally interpret intent. To fix this, the authors introduce the concept of aligned faithfulness and propose a new Predict-Select-Verify framework that uses contrastive highlights to guarantee explanations serve as genuine evidence for a model's decision.
What Problem Does This Paper Address?
When an AI model makes a text classification decision, how do we know why it made that choice? A popular approach in Natural Language Processing (NLP) is using "extractive rationales" or highlights—subsetting the input text to show the words that drove the decision.
The standard paradigm for generating these highlights is Select-Predict: a selector module picks a highlight, and a predictor module makes a decision based only on those highlighted words. Because the predictor never sees the rest of the text, this method is considered technically "faithful."
However, the authors identify a critical flaw: humans naturally project social intent onto explanations. We assume a highlight is either a comprehensive summary or direct evidence. The paper demonstrates that Select-Predict models can easily cheat this expectation through two failure modes:
- Trojan Explanations: The selector secretly encodes the final answer into arbitrary features of the highlight mask (e.g., the number of commas highlighted, or whether the highlight is at the beginning or end of the text), completely bypassing the semantic meaning of the words.
- The Dominant Selector: The selector makes the actual classification decision implicitly and manipulates the predictor into agreeing, hiding its true reasoning process.
Furthermore, forcing a model through a strict Select-Predict bottleneck inherently degrades its accuracy compared to looking at the full text.
Methods
To bridge this gap, the authors turn to social science literature on how humans communicate explanations. They define aligned faithfulness as an explanation that satisfies both causal attribution (it accurately reflects the model's mechanics) and social attribution (it matches human expectations of intent).
They propose an alternative causal pipeline called Predict-Select-Verify (PSV):
- Predict: The model makes a prediction based on the entire input text, preserving full performance.
- Select: A selector identifies a highlight.
- Verify: The system verifies that the predictor arrives at the exact same decision when evaluated on just the highlight.
To make these highlights meaningful to humans, the authors implement them as Contrastive Highlights. Instead of answering "Why class X?", they answer "Why class X rather than class Y (the foil)?"
If a user believes a certain text span supports class Y, the PSV model calculates the minimal additional words ($\text{h}_\Delta$) needed to flip the prediction back to the model's original choice, explicitly mapping out the decision boundary.
Datasets
The authors evaluated the vulnerabilities of Select-Predict models and the performance of their proposed framework across a wide array of standard NLP datasets:
- SST-2, SST-3, SST-5 (Stanford Sentiment Treebank)
- AGNews (News classification)
- IMDB (Movie reviews)
- Evidence Inference (Ev.Inf.) (Medical abstracts)
- 20News (Newsgroups discussion)
- Elec (Amazon Electronics reviews)
- MultiRC (Multi-sentence reading comprehension)
- Beer (Beer reviews)
Key Insights and Ideas
- Plausibility vs. Faithfulness is an Ethical Issue: Trying to force an explanation to look "plausible" (i.e., human-like) can incentivize a model to lie about its actual reasoning path, especially when it makes a mistake.
- Trojan Explanations are a Real Threat: The authors experimentally proved that existing select-predict architectures leak label information through mask metadata. Classifiers trained purely on the formatting of the highlights (like punctuation counts) could predict the model's final label at above-random chance.
- Eliminating the Performance Trade-off: Because the primary prediction in PSV happens on the full text, developers no longer have to choose between having a highly accurate model and an interpretable one.
Key Results
Proving the Trojan Vulnerability
The authors trained an RNN classifier to predict a model's final decision using only the binary highlight mask ($h$), completely blinding it to the text. As shown below, the mask alone retained substantial predictive power:
They also discovered that simple character statistics within the highlighted text (like counting commas or capital letters) allowed an MLP to predict the model's output significantly better than a baseline using full-text statistics:
Performance Degradation in Select-Predict
The paper quantifies the massive drop in accuracy that standard select-predict models face. Across multiple datasets, forcing the model through a highlight bottleneck caused error rates to spike drastically—in some cases, doubling or tripling the error:
By contrast, the proposed Predict-Select-Verify framework avoids this degradation entirely by decoupling the initial prediction from the highlight generation.
Limitations and Future Work
While Predict-Select-Verify resolves the alignment and performance issues, it introduces new computational challenges. Finding the absolute minimal contrastive highlight ($\text{h}_\Delta$) currently relies on a brute-force search or simple heuristics, which can scale poorly on long documents.
Additionally, the quality of a manual contrastive explanation relies on the user providing a meaningful counterfactual "foil." The authors note that extending this rigorous social alignment framework to continuous explanation formats—like attention maps, LIME, or SHAP heatmaps—remains an open and vital direction for future research.
Summary updated