In the quest to understand deep neural networks, researchers often rely on "explanations"—heatmaps or saliency scores that highlight which input features (like pixels or words) were most important for a prediction. However, these explanations are notoriously fragile. A tiny, imperceptible change to an image can cause a model to shift its "reasoning" entirely, even if the final classification remains the same.
Recent research has uncovered a fundamental link between a model’s adversarial robustness (its ability to resist noise) and its interpretability (the reliability of its explanations). The core intuition is simple: if a model is sensitive to noise, its gradients—the mathematical foundation of most explanations—will be noisy and chaotic. If a model is robust, its gradients become structured, meaningful, and stable.
The Fragility of Post-Hoc Explanations
Standard Explainable AI (XAI) methods often suffer from extreme sensitivity. Anh Totti Nguyen and colleagues demonstrated in SAM: The Sensitivity of Attribution Methods to Hyperparameters that even arbitrary choices, like a random seed or a patch size, can lead to wildly different interpretations of the same decision.
This fragility isn't just a nuisance; it's a security risk. Wei Huang 0035 and others developed the SAFARI framework to show that many XAI tools can be easily manipulated to produce "worst-case" explanation flips, where a model appears to be looking at the background when it is actually focusing on the object.
Robustness as a Prerequisite for Interpretability
The breakthrough in this field is the realization that we cannot have stable explanations without a stable model. Zeren Tan and Yang Tian argue in Robust Explanation for Free or At the Cost of Faithfulness that models trained for adversarial robustness naturally produce stable explanations "for free."
When a model is trained to ignore adversarial noise, it learns to rely on a smaller set of core, invariant features. This has two major effects:
- Cleaner Gradients: The gradient signals used for attribution become less sensitive to hyperparameter settings, as shown by Anh Totti Nguyen.
- Compactness: Min Wu and colleagues found in Efficiently Computing Compact Formal Explanations that robust models yield smaller, more focused "formal explanations" because the model has learned to ignore peripheral noise.
Training for Attribution Stability
If robustness improves explanations, can we flip the script? Can forcing a model to have stable explanations make the model itself more robust?
The answer appears to be yes. Jiefeng Chen introduced Robust Attribution Regularization (RAR) in Robust Attribution Regularization., which penalizes the model during training if its Integrated Gradients change too much under perturbation. Similarly, Fan Wang proposed the Integrated Gradient Regularizer (IGR) in Exploiting the Relationship Between Kendall's Rank Correlation and Cosine Similarity for Attribution Protection, which protects the rank-order of feature importance.
Both groups found that regularizing the "reasoning" of the model (the attributions) actually improves the model's overall adversarial accuracy. By forcing the model to maintain consistent explanations, we force it to develop more stable internal representations.
The Faithfulness Trade-off
While making explanations robust is desirable, there is a hidden danger: faithfulness. An explanation is "faithful" if it accurately reflects how the model actually works.
Zeren Tan warns that if we use post-hoc smoothing techniques to make an explanation look stable on a non-robust model, we might be creating a "robustness-faithfulness trade-off." The explanation might look pretty and stable to a human, but it no longer accurately reflects the chaotic, noise-sensitive reality of the underlying model.
To solve this, frameworks like REGEX, proposed by Baotian Hu in Towards Faithful Explanations for Text Classification, combine adversarial training with self-training to ensure that the model's internal attention mechanisms and its external feature attributions remain aligned and robust simultaneously.
Formalizing the Connection
This shift from subjective "visual appeal" to rigorous stability is best captured by the mathematical framework of $\delta$-interpretability. Amit Dhurandhar and colleagues argued in A Formal Framework to Characterize Interpretability of Procedures that a procedure is only truly interpretable if the information it transfers to a target (like a human) remains useful even under distribution shifts or adversarial attacks.
By treating robustness as a fundamental parameter of interpretability, the field is moving toward AI that is not just "explainable," but reliably and provably so.