Introduction
Continual Learning (CL) aims to enable AI models to learn a sequence of tasks over time without forgetting previously acquired knowledge—a challenge known as catastrophic forgetting. While sparse models offer a path toward computational efficiency, they typically necessitate expensive "prune-then-retrain" cycles that are impractical in sequential learning scenarios. EsaCL (Efficient Sharpness-aware Continual Learning) addresses this by introducing a method to achieve model sparsity and data efficiency simultaneously without the need for retraining.
Methodology
EsaCL introduces two primary innovations to streamline the continual learning process:
Sharpness-aware informed Pruning (SDP)
The core intuition behind SDP is the geometry of the loss landscape. Standard magnitude pruning often leads to significant performance drops because it may occur in "sharp" regions of the objective function. By utilizing Sharpness-aware Minimization (SAM) principles and Stochastic Frank-Wolfe (SFW) optimization, EsaCL encourages the model to converge to "flat" minima.
In a flat region, setting small weights to zero (pruning) acts as a minor perturbation that does not significantly alter the model's output. This allows for "one-shot" pruning during the training of each task, bypassing the need for a separate retraining phase.
Intelligent Data Selection (IDS)
To further enhance efficiency, EsaCL prunes the training data alongside the model weights. This is achieved through two sub-strategies:
- Support Example Selection (SSE): Identifies "hard" examples with high loss to help define the boundaries of the flat loss region.
- Redundant Example Elimination (ERE): Removes samples that have minimal influence on the model's gradients, accelerating training without sacrificing accuracy.
Experimental Results
The researchers evaluated EsaCL on standard benchmarks including Split CIFAR-10, Split CIFAR-100, and Tiny-ImageNet.
Performance and Efficiency
EsaCL demonstrated superior performance compared to established baselines. For instance, EsaCL-EWC outperformed standard EWC by 2% on CIFAR-10 and 3.5% on CIFAR-100. More importantly, it achieved these results with significantly lower resource requirements. Compared to SparCL, a leading efficient CL baseline, EsaCL maintained similar or better accuracy while utilizing substantially less memory—roughly half the footprint on CIFAR-100—by eliminating the need to store intermediate masks for every task.
As shown in the results, EsaCL remains robust even as a significant percentage of training data is removed, often outperforming SparCL at higher removal rates.
Computational Gains
The Intelligent Data Selection (IDS) strategy significantly reduced training FLOPs. On CIFAR-10, training costs were reduced from 0.46 to 0.33 scaled FLOPs while maintaining accuracy. Furthermore, the model showed stable performance across sparsity ratios of 60% to 80% without post-pruning retraining.
Limitations and Future Work
Despite its successes, EsaCL is sensitive to the choice of the perturbation radius ($\rho$). If $\rho$ is set too high, the model may fail to find an optimal flat region, leading to sharp performance declines. Additionally, the data selection strategy showed some sensitivity to noise in more complex datasets like Tiny-ImageNet. Future research will focus on developing more robust data subset selection methods and providing a deeper theoretical foundation for the effectiveness of flat-region optimization in continual learning.
Summary updated