When we deploy post-hoc explainable AI (XAI) methods—such as feature attributions, saliency maps, or counterfactuals—we expect them to reveal the inner workings of a machine learning model. We treat these explanations as a window into the model's decision-making process. However, a troubling reality has emerged across recent research: these windows are often highly distorted by the very tools we use to open them.
Many popular XAI methods exhibit severe hyperparameter sensitivity. Arbitrary choices made during the explanation process—such as random seeds, patch sizes, masking thresholds, or regularization constraints—can fundamentally alter the resulting explanation. If changing a random seed or a patch size completely flips which features are deemed "important," the explanation ceases to be a faithful reflection of the model. Instead, it becomes an artifact of the explanation algorithm's configuration noise.
The Causal Confounding of Explanations
To understand why this instability occurs, we must look at how explanations are generated. Many post-hoc explainers treat the model as a black box, perturbing inputs or optimizing a surrogate objective to approximate local behavior. In doing so, they introduce a suite of hyperparameters that act as causal interventions.
In On the Relationship Between Explanation and Prediction: A Causal View, Amir-Hossein Karimi et al. formalize this using causal inference. They demonstrate that in many high-performing networks, the generated explanations are heavily influenced by training configurations and optimization noise rather than the model's actual inference logic. When an explanation method is more sensitive to its own hyperparameters than to the model's predictions, it effectively decouples explanation from prediction.
Fragility Across Attribution and Search Methods
This sensitivity manifests across diverse modalities and algorithmic families. For instance, in computer vision, input perturbation methods rely on masking parts of an image to see how the model's output changes. Lukas Brunke et al. show in Evaluating Input Perturbation Methods for Interpreting CNNs and Saliency Map Comparison that choices like mask resolution and sampling probability drastically alter saliency maps, undermining reproducibility. Similarly, Zhiyu Zhu et al. note in Iterative Search Attribution for Deep Neural Networks that search-based attribution methods are highly sensitive to step sizes and scaling factors used to rank feature importance.
Even classical models and structured data are not immune. For example, Philipp Probst et al. demonstrate in Hyperparameters and Tuning Strategies for Random Forest that while hyperparameter tuning is routinely used to optimize predictive accuracy, its volatile impact on variable importance measures is frequently ignored, leading to unstable feature rankings.
In high-stakes clinical settings, Amin Nayebi et al. find in An Empirical Comparison of Explainable Artificial Intelligence Methods for Clinical Data: A Case Study on Traumatic Brain Injury that rule-based and counterfactual explanations suffer from low stability under minor parameter adjustments, threatening user trust where consistent reasoning is paramount.
The Meta-Evaluation Crisis and Auditing Vulnerabilities
If explanations are unstable, can we at least use evaluation metrics to filter out the bad ones? Unfortunately, the metrics themselves suffer from the exact same flaw.
Anna Hedström et al. introduce a meta-evaluation framework in The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantus to address how faithfulness metrics inconsistently rank explanation methods due to their own hyperparameter sensitivities (such as masking thresholds).
This lack of metric robustness creates a severe vulnerability for AI governance. As argued by Pratinav Seth et al. in Bridging the Gap in XAI-Why Reliable Metrics Matter for Explainability and Compliance, interpretability metrics can be deliberately tuned or exploited to generate misleadingly positive explanations. This allows for "fairwashing"—manipulating an explanation to look compliant or unbiased without changing the model's actual logic.
To combat this, Zihao Fu proposes a new metric called AXE in Evaluating the Ability of Explanations to Disambiguate Models in a Rashomon Set, which evaluates explanations based on how well their highlighted features predict the model's output, successfully detecting adversarial fairwashed models where traditional metrics fail.
Mitigating Instability: From Robust Models to Ensembles
How can we build explanations that we can actually trust? Researchers are pursuing several promising directions to mitigate hyperparameter sensitivity:
- Adversarial Model Robustness: In SAM: The Sensitivity of Attribution Methods to Hyperparameters, Anh Totti Nguyen et al. uncover a fascinating synergy: models trained to be robust against adversarial noise naturally yield much more stable and reliable explanations across varying hyperparameter choices.
- Hyperparameter Smoothing and Ensembling: Rather than searching for a single "perfect" hyperparameter setting, we can aggregate explanations across a range of configurations. Vadim Borisov demonstrates this in A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines, using Restricted Boltzmann Machines to extract a robust consensus from multiple noisy explanation maps generated under varying parameter scales.
- Algorithmic Reformulations: Other approaches replace fragile optimization loops with more stable mathematical formulations. For instance, Arash Rahnama et al. propose AXAI in An Adversarial Approach for Explaining the Predictions of Deep Neural Networks, which repurposes adversarial attacks to pinpoint influential features faster and more stably than LIME or SHAP. Similarly, Yi Sui et al. develop RPS-LJE in Representer Point Selection via Local Jacobian Expansion for Post-hoc Classifier Explanation of Deep Neural Networks and Ensemble Models to provide stable data attribution without the extreme sensitivity to regularization hyperparameters that plagues traditional methods.
The shift toward analyzing hyperparameter sensitivity marks a maturity milestone for Explainable AI. By recognizing that an explanation can be an artifact of its own tuning parameters rather than the underlying model, the field is moving away from fragile, ad-hoc visualizations toward robust, mathematically verified diagnostic tools.
Key Foundational Work
Hyperparameters and Tuning Strategies for Random Forest (2018) — contributed 2 concept elements to this direction.
Layer-Wise Relevance Propagation for Explaining Deep Neural Network Decisions in MRI-Based Alzheimer's Disease Classification (2019) — contributed 1 concept elements to this direction.
Iterative and Adaptive Sampling with Spatial Attention for Black-Box Model Explanations (2019) — contributed 1 concept elements to this direction.