Introduction
In the field of Continual Learning (CL), models are tasked with learning from a non-stationary stream of data where new classes or tasks appear sequentially. The primary challenge in this domain is "catastrophic forgetting," a phenomenon where the acquisition of new knowledge causes the model to lose its ability to perform previously learned tasks. While significant research has focused on maintaining predictive accuracy, the stability of the underlying logic—how and why a model makes a decision—has remained largely unexplored.
This paper addresses this gap by introducing a formal protocol and quantitative metrics to measure "explanation drift." If a model correctly identifies a digit today but relies on entirely different features than it did yesterday, its reliability is questionable. The authors propose that for a model to be truly robust in a continual setting, its explanations must remain consistent with a "ground truth" established by a model trained on all data simultaneously.
Methodology: Quantifying Explanation Drift
The authors focus on SHAP (SHapley Additive exPlanations), a game-theoretic approach that assigns importance values to input features. To evaluate how explanations evolve during continual learning, they compare the SHAP values of a CL-trained model against a Joint Training baseline (a model trained offline on the full dataset).
To quantify the distance between these explanations, the paper introduces two novel metrics:
- Metric $M$: This measures the squared difference between the sum of positive SHAP values of the CL model and the Joint Training model. It provides a global view of how much the "positive evidence" for a class shifts.
- Metric $M_{pool}$: Designed specifically for image data, this metric applies 2D average pooling to the SHAP maps before comparison. This accounts for spatial locality, ensuring that minor pixel-level shifts in importance do not unfairly penalize the model if the general area of interest remains the same.
Evaluation Protocol
The protocol evaluates explanations for all possible classes, not just the one predicted by the model. This "class-agnostic" approach is vital in CL, as it allows researchers to understand the logic behind misclassifications and track how the internal representation of a class changes even before the model officially "forgets" it.
Experimental Setup
The researchers tested their protocol across three distinct benchmarks:
- Split MNIST & Split CIFAR-10: Standard computer vision tasks using ResNet and convolutional architectures.
- Synthetic Speech Commands (SSC): An audio sequence task processed into Mel features, requiring recurrent architectures like LSTMs.
They compared three common CL strategies:
- Naive: Simple fine-tuning on new data (the baseline for forgetting).
- Experience Replay (ER): Maintaining a small buffer of past examples to periodically rehearse.
- Gradient-based Sample Selection (GSS): An advanced replay strategy that selects buffer samples based on their impact on the gradient space.
Key Findings
1. Stability in Vision Tasks
For feedforward and convolutional models, the results were intuitive: anti-forgetting strategies like ER and GSS successfully stabilized both accuracy and explanations. In the MNIST benchmark, for instance, the $M_{pool}$ metric for the Naive strategy was approximately double that of the Replay strategies, indicating that rehearsal effectively keeps the model's "attention" on the correct features over time.
2. The RNN Paradox
A surprising discovery emerged in the sequence-based SSC benchmark. While Experience Replay improved the accuracy of LSTMs, it failed to stabilize their explanations. In some cases, the Naive approach actually exhibited lower explanation drift for later classes than the Replay strategies did. This suggests that in fully-trained Recurrent Neural Networks (RNNs), the process of updating recurrent weights to accommodate new tasks inherently destabilizes the feature importance of previous tasks, even if the final prediction remains correct.
3. Echo State Networks as a Solution
To combat the instability found in RNNs, the authors explored Echo State Networks (ESNs). In an ESN, the recurrent layer (the "reservoir") is fixed with random weights, and only the output layer is trained. The experiments revealed that ESNs achieved significantly lower explanation drift ($M$) across almost all classes compared to LSTMs, without a loss in accuracy. This suggests that fixing the recurrent dynamics is a powerful way to ensure "explanation permanence" in sequential data processing.
Conclusion and Future Work
This research highlights that accuracy is an incomplete metric for evaluating Continual Learning. By providing a protocol to measure explanation drift, the authors reveal that different architectures (CNNs vs. RNNs) react differently to standard anti-forgetting techniques.
The study opens several avenues for future work, including the development of new CL loss functions that explicitly penalize explanation drift. Furthermore, the discovery that ESNs provide naturally stable explanations suggests that reservoir computing may hold unique advantages for trustworthy AI in non-stationary environments. Future research will likely extend these metrics to other XAI techniques beyond SHAP and explore larger-scale datasets.
Summary updated