Explainable AI (XAI) promises to open the "black box" of deep learning, offering practitioners a window into the reasoning of complex models. However, a growing body of research suggests that these windows are often distorted. The very tools we use to explain models—such as saliency maps, feature attributions, and dataset meta-features—are themselves highly sensitive to internal hyperparameters. This sensitivity creates a "meta-interpretability" problem: it is often unclear whether an explanation reflects the fundamental logic of the model or is merely an artifact of the explanation tool's configuration.
The Fragility of Feature Attribution
The most common way to explain a vision model is through feature attribution, which highlights the pixels most responsible for a prediction. Methods like LIME, SHAP, and Integrated Gradients are widely used, yet Anh Totti Nguyen, Chirag Agarwal, and Anh Nguyen 0002 demonstrate in SAM: The Sensitivity of Attribution Methods to Hyperparameters that these methods are alarmingly sensitive to arbitrary choices. A change in a random seed or the size of a perturbation patch can fundamentally alter which features are deemed "important."
This instability is not limited to vision. In speech-to-text models, Dennis Fucci and colleagues found that the effectiveness of input masking in their SPES framework depends heavily on heuristic hyperparameters like clustering counts. Similarly, Bhavan Vasu and Chengjiang Long note in IASSA that improperly tuned window sizes can introduce structural artifacts, such as grid-like patterns, that obscure the true relationship between input and output.
The Meta-Evaluation Problem
If explanations are unstable, can we at least use quantitative metrics to rank them? Research into "meta-evaluation"—evaluating the evaluators—suggests this is equally difficult. Anna Hedström and the creators of the Quantus toolkit have shown that metrics designed to measure "faithfulness" (how well an explanation tracks model behavior) are themselves sensitive to implementation choices.
For example, a faithfulness metric might involve masking the "top 10%" of features and measuring the drop in model confidence. However, as noted in The Meta-Evaluation Problem in Explainable AI, the choice of what value to use for "masking" (e.g., black pixels, mean color, or noise) can completely flip the ranking of which XAI method appears superior. This circular dependency means that without a "ground truth" for explanations, researchers are often just tuning hyperparameters until the explanation "looks right," a process that Lukas Brunke and others warn undermines the reliability of auditing black-box models.
Beyond Post-Hoc Saliency
The sensitivity issue extends beyond simple heatmaps into more structural forms of interpretability:
- Mechanistic Interpretability: Even when trying to find "circuits" (specific sub-networks responsible for behaviors), the results are unstable. Maxime Méloux and Maxime Peyrard argue in Mechanistic Interpretability as Statistical Estimation that discovered mechanisms are often artifacts of experimental design—such as the specific noise level used in causal interventions—rather than universal properties of the model.
- Dataset Characterization: In meta-learning, Adriano Rívolli and Joaquin Vanschoren point out that meta-features (like the depth of a decision tree used to describe a dataset) are often treated as hidden variables. Failure to report these internal settings creates a reproducibility gap in characterizing classification datasets.
- Clustering Trajectories: Even in unsupervised learning, Thomas Seidl and Daniyal Kazempour suggest that we must look at "Data Point Entanglement"—how points move during iteration—to distinguish fundamental data structure from hyperparameter artifacts.
Toward Robust and Automated Interpretability
To move past this "Wild West" of manual tuning, the field is shifting toward two solutions: robustness and automation.
Amir-Hossein Karimi and Been Kim propose using causal inference to separate the influence of hyperparameters from the model's actual prediction logic. They find that in many high-performing models, the explanation is more a function of the training configuration than the inference process.
Meanwhile, frameworks like PnPXAI, developed by Seongun Kim and Jaesik Choi, attempt to automate the selection and optimization of explanation methods. By programmatically recommending the best method for a given architecture and data modality, these tools aim to remove the human-in-the-loop bias that currently plagues XAI. As Christian A. Scholbeck argues, interpretability must be treated as a subset of Sensitivity Analysis, a mature field that provides the formal tools needed to track how variations in the pipeline flow down into the final explanation.