Abstract
Explainable AI methods facilitate the understanding of model behaviour, yet, small, imperceptible perturbations to inputs can vastly distort explanations. As these explanations are typically evaluated holistically, before model deployment, it is difficult to assess when a particular explanation is trustworthy. Some studies have tried to create confidence estimators for explanations, but none have investigated an existing link between uncertainty and explanation quality. We artificially simulate epistemic uncertainty in text input by introducing noise at inference time. In this large-scale empirical study, we insert different levels of noise perturbations and measure the effect on the output of pre-trained language models and different uncertainty metrics. Realistic perturbations have minimal effect on performance and explanations, yet masking has a drastic effect. We find that high uncertainty doesn't necessarily imply low explanation plausibility; the correlation between the two metrics can be moderately positive when noise is exposed during the training process. This suggests that noise-augmented models may be better at identifying salient tokens when uncertain. Furthermore, when predictive and epistemic uncertainty measures are over-confident, the robustness of a saliency map to perturbation can indicate model stability issues. Integrated Gradients shows the overall greatest robustness to perturbation, while still showing model-specific patterns in performance; however, this phenomenon is limited to smaller Transformer-based language models.
Introduction
Explainable AI (XAI) tools, such as saliency maps, are intended to provide transparency into model decision-making by highlighting the most influential input features. However, these explanations are often brittle: minor, human-imperceptible changes to input text can lead to significant distortions in the resulting explanations, even if the model's final prediction remains unchanged. This lack of robustness makes it difficult to determine when an explanation can be trusted during inference. This study investigates whether standard uncertainty metrics—which quantify a model's "confidence"—can serve as reliable proxies for explanation quality.
Methodology
The researchers conducted a large-scale empirical study using five Transformer-based language models: BERT, RoBERTa, ELECTRA, GPT-2, and OPT-350M. These models were evaluated across three text classification tasks: SST-2 (movie reviews), SemEval 2013 (Twitter sentiment), and HateXplain (hate speech detection).
The study introduced several novel experimental dimensions:
- Perturbation Hierarchies: Noise was applied based on three strategies: Random, Human (targeting tokens identified as important by human annotators), and Gradient (targeting tokens identified as salient by the model's gradients).
- Diverse Noise Types: Seven noise types were tested, ranging from semantic-preserving (synonyms, typos) to structural (token masking, l33t speak).
- Uncertainty and XAI Metrics: The study tracked both Predictive Uncertainty (softmax entropy) and Epistemic Uncertainty (MC Dropout). Four gradient-based explanation methods were compared: SmoothGrad, Guided Backpropagation, InputXGradients, and Integrated Gradients (IG).
Key Results and Analysis
The Impact of Salient Tokens
The study found that perturbing tokens identified as important by humans (Human hierarchy) had a significantly more detrimental effect on both model performance and explanation plausibility compared to random perturbations.
As shown in Figure 1, for the RoBERTa model on SST-2, the Human hierarchy leads to a sharper decline in accuracy and a more pronounced increase in uncertainty compared to other patterns. Interestingly, RoBERTa often exhibited over-confidence, with uncertainty metrics remaining relatively low even as accuracy dropped significantly.
Robustness of Explanation Methods
Among the evaluated XAI methods, Integrated Gradients (IG) consistently demonstrated the highest robustness to noise in smaller Transformer models. It maintained more stable saliency maps even as input quality degraded.
Figure 3 illustrates the robustness across models at a 10% perturbation level. While methods like Guided Backpropagation often showed high sensitivity, IG (represented by the green area) generally covered a larger portion of the radar charts, indicating better stability across various noise types.
The Uncertainty-Plausibility Paradox
A key finding is the "uncertainty-plausibility" paradox. In "clean" datasets like SST-2, higher uncertainty typically correlated with lower explanation quality. However, in the noisier HateXplain dataset—which contains social media text—the correlation was often positive. This suggests that models exposed to noise during training may become more adept at identifying salient features when encountering ambiguous or noisy inputs at test time.
Conclusion
This research highlights that uncertainty is not a perfect proxy for explanation quality; the correlation is often low-to-moderate and context-dependent. While structural noise like masking drastically distorts both confidence and saliency, semantic noise has a negligible effect. The study suggests that the robustness of a saliency map to minor perturbations can serve as a valuable warning sign for model instability, especially when internal uncertainty metrics are over-confident. Future work should extend these findings to larger-scale models and generative tasks.
Summary updated