Continual Learning: Forget-free Winning Subnetworks for Video Representations

Haeyong Kang, Jaehong Yoon, Sung Ju Hwang, Chang D. Yoo

Abstract

Artificial Intelligence systems often struggle with catastrophic forgetting, where learning new tasks degrades performance on previously acquired knowledge. This paper introduces a "forget-free" continual learning framework that identifies task-specific "winning subnetworks" (WSN) within a single dense model. By freezing weights associated with prior tasks, the model eliminates the need for expensive data replay or architectural expansion. Furthermore, the integration of a novel Fourier Subneural Operator (FSO) allows the model to capture global periodic signals, significantly improving performance in Video Incremental Learning (VIL). Our approach achieves state-of-the-art results across diverse image and video benchmarks with zero backward transfer.

Introduction

The challenge of continual learning (CL) is to enable models to learn a sequence of tasks without forgetting. Traditional approaches either store old data (rehearsal-based) or grow the network size (architecture-based), both of which have significant drawbacks in terms of memory, privacy, and scalability. This research focuses on building a memory-efficient, fixed-capacity model capable of handling complex data types like video, where standard spatial convolutions often fail to capture long-range temporal dependencies.

Methodology: Winning Subnetworks and Fourier Operators

The proposed framework is built upon the Lottery Ticket Hypothesis, which posits that dense networks contain sparse subnetworks capable of matching the performance of the full model.

Winning Subnetworks (WSN)

For each incoming task, the model identifies a sparse set of weights—a "winning ticket." Once a task is learned, its corresponding weights are frozen. Subsequent tasks are trained using only the remaining available weights. This mechanism ensures mathematically zero forgetting (zero backward transfer) in task-incremental settings.

Comparison of continual learning paradigms: (a) Fixed Backbone, (b) Biased Transfer, (c) Selective Reuse Expansion beyond Dense Network, and (d) the proposed Selective Reuse Expansion within Network (WSN).

Fourier Subneural Operator (FSO)

To better represent video data, the authors introduce the FSO. Unlike standard layers that operate in the spatial domain, the FSO transforms features into the frequency domain. This allows the identification of "periodic winning tickets," which are more effective at encoding global structures and temporal dynamics in videos. This is particularly effective in lower layers for video representation to encode base implicit representations.

Soft-Subnetwork (SoftNet) and Optimization

To address Few-Shot Class Incremental Learning (FSCIL), where data is scarce, the framework employs "soft" regularized masks instead of binary ones to prevent overfitting. The non-differentiable masking process is optimized using a Straight-Through Estimator (STE), allowing for end-to-end gradient-based training of the subnetwork structure.

Experimental Results

The framework was evaluated on standard image classification benchmarks (CIFAR-100, ImageNet-1K) and video datasets (UVG8, UVG17).

  • Accuracy and Forgetting: On CIFAR-100, the WSN+FSO combination reached 79.00% accuracy, surpassing baselines like GPM (73.18%) and SupSup (75.47%). Crucially, the Backward Transfer (BWT) remained at 0.0 across all task-incremental experiments.
  • Video Reconstruction: In VIL tasks, the FSO-enhanced model significantly outperformed standard Neural Implicit Representations (NeRV) in terms of Peak Signal-to-Noise Ratio (PSNR), indicating superior video reconstruction fidelity.
  • Efficiency: The model's robustness was further demonstrated through 8-bit quantization, which maintained performance levels similar to 32-bit versions, highlighting its suitability for edge deployment.

Discussion and Key Insights

A primary insight is that frequency-domain operations are vital for video; FSO helps re-inject global structural information that deep CNNs often lose. Additionally, the sparsity inherent in WSNs acts as a regularizer, leading the model to "flatter minima" in the loss landscape, which improves generalization.

While the fixed-capacity backbone provides a "forget-free" guarantee, it does introduce a capacity ceiling. As the number of tasks grows, the pool of available weights diminishes. Future research may explore selective weight updates to allow for positive backward transfer while maintaining stability.

Conclusion

By combining the efficiency of winning subnetworks with the global representational power of Fourier operators, this framework provides a robust solution for forget-free continual learning in both image and video domains.

Summary updated

BibTeX
@article{kang2025continual,
  author = {Haeyong Kang and Jaehong Yoon and Sung Ju Hwang and Chang D. Yoo},
  title = {{Continual Learning: Forget-free Winning Subnetworks for Video Representations}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2024.3518588},
}
Figure