Abstract
As Large Language Models (LLMs) are increasingly deployed in high-stakes environments, the reliability of their explanations (XAI) and uncertainty estimates becomes paramount. This study investigates the "brittleness" of these systems by analyzing how various forms of input noise affect five diverse language models. Our findings reveal that while structural noise significantly degrades performance, high uncertainty does not always signal unreliable explanations. Notably, in datasets characterized by natural noise, models can produce more plausible explanations even when their predictive uncertainty is high.
Introduction
Trust in AI systems currently rests on two pillars: Uncertainty Estimation (quantifying model confidence) and eXplainable AI (XAI) (providing rationales for predictions). However, XAI techniques are notoriously sensitive to minor input perturbations, such as typos or synonym swaps. This instability raises a critical question: Can a model's reported uncertainty serve as a proxy for the trustworthiness of its explanations? This paper explores the link between uncertainty metrics and explanation quality across different architectures and noise conditions.
Methodology
The researchers conducted an extensive empirical study using a noise injection framework across five models: BERT, RoBERTa, ELECTRA, GPT-2, and OPT-350M. The study involved:
- Perturbation Types: Seven types of noise were applied, ranging from character-level swaps and "butterfingers" typos to structural changes like
l33tspeak and token masking. - Targeting Hierarchies: Noise was injected based on random selection, human-annotated saliency, and gradient-based importance.
- XAI and Uncertainty Metrics: The team compared four gradient-based explanation methods (SmoothGrad, Guided Backpropagation, InputXGradients, and Integrated Gradients) against predictive uncertainty (softmax entropy) and epistemic uncertainty (MC Dropout entropy).
Experiments were conducted on three sentiment and hate speech classification datasets: SST-2, SemEval 2013, and HateXplain.
The Impact of Noise on Performance and Explanations
Not all noise impacts models equally. While "realistic" perturbations like synonym swaps have minimal effects on accuracy, structural noise such as l33t speak or [MASK] tokens causes drastic performance drops.
As shown in the figure above, as the percentage of text perturbed increases, accuracy declines while uncertainty metrics generally rise. However, a key observation is the phenomenon of "overconfidence": in high-noise scenarios (like l33t speak), uncertainty metrics often fail to rise proportionally to the drop in accuracy, indicating a failure mode where models confidently provide incorrect outputs.
XAI Robustness and the Uncertainty Paradox
A central finding of the study is the "Uncertainty-Plausibility Paradox." In cleaner datasets like SST-2, there is a low negative correlation between uncertainty and explanation quality. However, in noisier datasets like HateXplain, which contains slang and social media artifacts, there is a positive correlation. This suggests that models trained on diverse, noisy data may become more adept at identifying salient features specifically when they encounter ambiguous or difficult inputs.
Regarding the XAI methods themselves, Integrated Gradients (IG) emerged as the most robust technique for smaller Transformer models, maintaining higher explanation plausibility under character-level noise compared to other methods.
Conclusion and Future Directions
This research highlights that human intuition remains a vital signal; perturbing words that humans find important has a much more devastating effect on model stability than random noise. While Integrated Gradients stands out for its robustness, the study also underscores the limitations of current uncertainty metrics in detecting out-of-distribution failures.
Future work should expand this investigation to larger models (exceeding 1B parameters) and more complex tasks like summarization. Additionally, there is a clear need for real-time metrics that can estimate explanation quality at the data-point level without requiring human-annotated ground truths.
Summary updated