Aligning Faithful Interpretations with their Social Attribution

Alon Jacovi, Yoav Goldberg

Abstract

In natural language processing, "faithfulness" is the requirement that an explanation accurately reflects the internal reasoning of a model. However, current extractive rationale models often suffer from a misalignment between their causal mechanisms and the social intent humans attribute to highlights. This paper identifies two primary failure modes—Trojan explanations and dominant selectors—and proposes the Predict-Select-Verify (PSV) framework. By decoupling prediction from selection and adding a verification step, PSV ensures that textual highlights serve as genuine evidence while maintaining the performance of full-text models.

The Problem: Misaligned Faithfulness

In NLP, faithfulness usually refers to how well an explanation represents a model's internal reasoning. A common approach is the Select-Predict (SP) pipeline, where a "selector" extracts textual highlights (rationales) and a "predictor" makes a decision based solely on those highlights.

The authors argue that this pipeline is prone to two major failures:

  1. Trojan Explanations: The model "cheats" by encoding the answer in the highlight's metadata (e.g., length or position) rather than its semantic content. For instance, a model might learn that a highlight at the beginning of a text always signifies a positive sentiment, regardless of the words used.
  2. The Dominant Selector: The selector effectively makes the decision in secret and then passes a "plausible" highlight to the predictor to justify it. In this case, the highlight is not the reason for the decision but a tool to manipulate the output.

These issues stem from a lack of Aligned Faithfulness: the causal chain of the model does not match the social intent—such as providing evidence—that a human observer expects from an explanation.

Example of textual highlights as explanations. Different selection strategies (a vs b) can result in the same prediction, highlighting the potential for 'Trojan' signals where the selection process itself encodes the answer.

The Predict-Select-Verify (PSV) Framework

To address these misalignments, the authors propose shifting from the standard Select-Predict model to a Predict-Select-Verify (PSV) framework:

  • Predict: The model first makes a prediction based on the entire input text, ensuring it uses all available information and avoiding the performance loss common in SP models.
  • Select: A selector identifies a highlight that supports the pre-existing prediction.
  • Verify: The system confirms that the predictor would reach the same conclusion if it only saw the selected highlight.

This framework utilizes Contrastive Explanations, asking "Why did the model predict X instead of Y?" and employs masked language models (like BERT) to handle both full text and partial highlights naturally.

Key Results and Insights

The authors evaluated their approach, implemented as FRESH, against standard benchmarks including SST, IMDB, and AG News. They confirmed the existence of "Trojan" signals in existing models by showing that a simple classifier could often predict labels from the highlight mask alone, without seeing any words.

Crucially, the PSV approach avoids the "faithfulness tax"—the drop in accuracy often seen in Select-Predict models. Because the primary prediction is not bottlenecked by the selection process, FRESH maintains performance levels comparable to full-text models.

Performance comparison of the FRESH framework against full-text baselines and existing Select-Predict models across various datasets.

Conclusion

Explanations are social acts. By forcing the prediction to occur before the selection, the PSV framework ensures that highlights act as supporting evidence rather than hidden decision-makers. This alignment between causal reasoning and social attribution is a critical step toward more ethical and interpretable AI.

Summary updated

BibTeX
@article{jacovi2021aligning,
  author = {Alon Jacovi and Yoav Goldberg},
  title = {{Aligning Faithful Interpretations with their Social Attribution}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00367},
}
Figure