Abstract
As machine learning models become increasingly prevalent in time series applications, Explainable Artificial Intelligence (XAI) methods are essential for understanding their predictions. Feature attribution methods aim to identify which input features contribute most to a prediction, typically evaluated using perturbation-based metrics. This paper reveals previously overlooked class-dependent effects in these metrics: they show varying effectiveness across classes, often reflecting learned model biases rather than intrinsic attribution quality. We propose an evaluation framework with a class-aware penalty term to account for these effects, offering a more robust assessment of feature attributions. Code and results are available at github.com/gregorbaer/class-perturbation-effects.
The Problem: Biased Evaluation in XAI
In Explainable AI (XAI), feature attribution methods highlight which parts of an input—such as specific time steps in a time series—drive a model's prediction. Since "ground truth" explanations are rarely available, researchers rely on perturbation analysis for evaluation. The core assumption is that if an XAI method correctly identifies important features, perturbing or "destroying" those features should significantly decrease the model's confidence in its prediction.
However, researchers have noticed that uniform perturbation strategies often fail to affect large portions of a dataset equally across different classes. For instance, if a model associates a value of "zero" with a specific class, replacing features with zeros might inadvertently reinforce that prediction instead of disrupting it. This paper systematically investigates whether current XAI evaluation metrics are inadvertently measuring these class-specific model biases rather than the actual quality of the feature attributions.
Methodology
The authors evaluated feature attributions using the Degradation Score (DS), which compares two perturbation curves:
- MoRF (Most Relevant First): Perturbing features in descending order of importance.
- LeRF (Least Relevant First): Perturbing features in ascending order of importance.
A high-quality attribution method should result in a rapid drop in accuracy under MoRF and a slow drop under LeRF. The area between these curves defines the Degradation Score.
To address class biases, the authors introduced the Class-Adjusted Degradation Score ($DS_c$). This novel framework subtracts a penalty term ($\Delta$) from the standard aggregate metric. This penalty quantifies the absolute difference in attribution performance between classes, punishing methods or strategies that only work well for specific classes while failing for others.
Datasets and Models
The study utilized four univariate time series datasets from the UCR Archive: FordA, FordB, Wafer, and ElectricDevices (multiclass). Evaluations were performed across five attribution methods (Gradients, Integrated Gradients, SmoothGrad, Gradient SHAP, and Feature Occlusion) using ResNet and InceptionTime architectures.
Key Insights
- The "Best" Strategies are Often the Most Biased: Perturbation strategies that appear highly effective in aggregate (like replacing values with Zero or the Subsequence Mean) often suffer from the most pronounced class-dependent effects. Their high scores are frequently an illusion driven by massive success on one class and near-zero impact on another.
- Evaluation Reflects Model Bias: The findings suggest that perturbation-based evaluation is deeply entangled with the model's learned representations. Metrics often measure how the model reacts to out-of-distribution data rather than just the quality of the explainer.
- Direction and Magnitude Matter: When testing constant-value perturbations, the sign and magnitude of the value heavily influenced the outcome. Models react asymmetrically to the direction of the perturbation in the feature space, further complicating evaluation.
Key Results
- Metric Collapse: When applying the proposed class-adjusted metric ($DS_c$), the scores of previously top-performing perturbation strategies plummeted, often shifting toward zero or becoming negative. This confirms that their perceived effectiveness was largely due to class bias.
- Attribution Method Rankings: Feature Occlusion (FO), Integrated Gradients (IG), and Gradient SHAP (GS) generally achieved the highest Degradation Scores across datasets, indicating better feature importance identification compared to standard Gradients (GR) and SmoothGrad (SG).
- Dataset Sensitivity: Optimal perturbation constants varied significantly by dataset. For example, in the FordA and FordB datasets, a constant perturbation of 0.5 often yielded the best class-adjusted performance, but this did not generalize to all tasks.
Conclusion
This research highlights a critical blind spot in how we evaluate XAI for time series. By accounting for class-dependent effects, the proposed framework ensures that evaluations reflect true explanation quality rather than exploiting model biases. Future work should focus on developing adaptive evaluation strategies that select perturbation methods on a per-instance basis to more accurately validate attributions across diverse data distributions.
Summary updated