Aligning Model Feature Attribution with Human Rationales

310 concepts203 papers4 proposals

Deep neural networks are notorious for "cheating." Because they are optimized solely to minimize prediction error on a training set, they routinely latch onto spurious correlations—coincidental patterns in the data that happen to correlate with the target label but lack any causal relationship. For instance, a computer vision model might classify a skin lesion as malignant simply because there is a ruler or a band-aid in the image (Singh et al.). Similarly, a text classifier might flag a sentence as hate speech purely because it contains neutral identity terms like "Muslim" or "gay," ignoring the actual context (Kennedy et al.).

Traditionally, eXplainable AI (XAI) tools like feature attributions, saliency maps, and attention visualizations have been used as post-hoc diagnostics to uncover these embarrassing failures. However, a growing body of research shifts explainability from a passive post-hoc diagnostic tool into an active training-time supervisory signal. By penalizing models when their internal reasoning diverges from human domain knowledge, this paradigm forces neural networks to be right for the right reasons.

Explanation as a Regularizer

The core mechanism of this research direction is to modify the model's optimization landscape. Instead of training a model exclusively on task labels (e.g., cross-entropy loss), the training objective incorporates an auxiliary loss term that evaluates how the model reached its conclusion:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \lambda \mathcal{L}_{\text{explanation}}$$

Here, $\mathcal{L}_{\text{explanation}}$ measures the discrepancy between the model's internal feature importance scores and a target explanation mask—such as human-annotated "rationales" or expert-defined regions of interest. The hyperparameter $\lambda$ controls the balance between standard task accuracy and explanation alignment.

By treating human rationales as an attribution prior during training, the authors of Human Rationales as Attribution Priors for Explainable Stance Detection demonstrate that nudging a model's internal feature importance to align with human reasoning significantly improves explainability in data-scarce and zero-shot settings without sacrificing predictive accuracy. This joint training objective transforms post-hoc diagnostics into an active training signal, ensuring that predictions are based on valid causal factors rather than dataset artifacts (Liu et al.).

Mathematical Mechanisms of Alignment

How do you mathematically penalize a model's "reasoning"? Researchers have developed several distinct strategies depending on the model architecture and the type of attribution metric being targeted.

Attention Supervision

For models that rely on attention mechanisms (like Transformers), the simplest approach is to directly supervise the attention weights. In HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection, Mathew et al. integrate human-provided rationales into training by explicitly penalizing the divergence between a model's internal attention weights and ground-truth text spans.

Similarly, Stacey uses a secondary loss term (such as Mean Squared Error) to align attention weights with human-annotated importance scores in Natural Language Inference (NLI). This mechanism forces the model to utilize the context of the premise, effectively mitigating "hypothesis-only bias" where models ignore the context and rely solely on patterns within the target sentence.

Gradient and Post-Hoc Attribution Penalization

Because raw attention weights do not always faithfully represent feature importance, many approaches penalize the actual gradients or post-hoc attribution scores computed during the backward pass.

  • Contextual Decomposition Explanation Penalization (CDEP): Introduced by Singh et al., this method penalizes the discrepancy between a model's internal feature importance scores (derived via contextual decomposition) and predefined ground-truth explanation masks. This forces the model to align its decision-making logic with human prior knowledge or domain-specific constraints during optimization.
  • Gradient Correction Modules: In aspect-based sentiment classification, Zhou et al. introduce a secondary loss term that penalizes discrepancies between a model's internal saliency scores and human-provided opinion words, forcing the model's internal logic to align with known relevant features.

Gradient Boosting

Rather than adding a soft loss penalty, Errica et al. propose an "error boosting" technique where human-provided rationales are incorporated by directly multiplying the gradients of specific tokens identified as important by annotators. This approach aggressively forces the model's internal representations to prioritize the most informative features identified by domain experts, proving highly robust in low-resource settings.

What to Supervise: Positive vs. Negative Feedback

When designing an explanation-guided training loop, a critical question arises: Should we tell the model what to focus on (positive feedback), or what to ignore (negative feedback)?

In an exploration of Explanatory Interactive Learning (XIL), Hagos et al. revealed a powerful asymmetry: telling a model what to ignore (spurious features) is significantly more effective for improving both classification accuracy and explanation quality than telling it what to focus on (valid features). Penalizing the model for relying on specific spatial regions or tokens identified as irrelevant forces the network to actively search for more robust, alternative causal features that generalize better to unseen data.

Why It Works: Smoothing and Regularization

Forcing a model to match human explanations does more than make the model "look pretty"—it fundamentally alters the optimization process to improve data efficiency and out-of-distribution generalization.

Data Efficiency via Ranking Constraints

Training deep models usually requires massive datasets. However, human rationales provide a dense supervisory signal that contains far more information than a simple binary label. By framing human rationales as ranking constraints in the loss function—where the model is forced to be more confident on evidence-rich snippets than on versions of the text where that evidence is removed—models can achieve state-of-the-art performance with up to 3x less training data than standard fine-tuning (Wang).

Smoothing the Optimization Landscape

In cooperative rationalization frameworks (where an explanation generator and a predictor are trained jointly), models frequently fall into an "interlocking trap" where both components reinforce each other's sub-optimal choices.

Yu et al. demonstrate that using a soft-attention predictor to guide a hard-selection generator smooths the optimization landscape. This guidance prevents the model from getting stuck in local minima, allowing it to capture features that are both mathematically faithful to the network's inner workings and semantically relevant to humans.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 219 papers
  2. Version 1 · 2026-10-04 · 214 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_supervised_attention_0cd680be,
  title = {supervised attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/supervised-attention/rdr_0cd680be43134e5abb149456408ab9a4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_supervised_attention_0cd680be_v2,
  title = {supervised attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/supervised-attention/rdr_0cd680be43134e5abb149456408ab9a4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake