Continual learning requires deep neural networks to master new tasks sequentially without forgetting previously acquired knowledge. In static image classification, the gold standard for preventing this "catastrophic forgetting" is rehearsal (or experience replay): keeping a small buffer of past images and interleaving them with new data during training.
However, when we transition from static images to videos, this paradigm breaks down. As noted by Park et al. and Spampinato et al., a single video clip contains dozens or hundreds of high-dimensional spatio-temporal frames. Storing even a modest number of raw video exemplars quickly exhausts deployment memory, creating a severe scalability bottleneck for real-world applications like robotics or edge computing.
To address this challenge, a new research direction focuses on Memory-Efficient Video Continual Learning. The goal is to retain the critical temporal and semantic information of past video streams while drastically reducing or entirely eliminating the physical storage overhead of raw video frames.
Redefining the Memory Budget: From Clips to Frames
In traditional image-based continual learning, memory constraints are typically measured by the number of instances saved. In the video domain, this metric is easily exploited; a model could artificially maximize its exemplar count by selectively storing incredibly short video clips.
To establish a rigorous evaluation standard, Villa et al. introduced vCLIMB, a benchmark asserting that video memory capacity must be measured by the total number of frames stored, rather than the number of clips. Under a strict frame budget, researchers face a fundamental trade-off: is it better to store a few complete, high-resolution video clips, or a sparse selection of frames across many different videos?
Temporal Condensation: Prioritizing Diversity Over Resolution
A key intuition emerging in this field is that instance diversity matters more than temporal resolution when preventing forgetting. If a replay buffer only contains a few full video clips, the model tends to overfit to the specific backgrounds or actors in those few examples, failing to retain the broader class distribution.
To exploit this, Alssum et al. proposed the SMILE framework, demonstrating that storing just one representative frame per video is significantly more effective than storing multiple frames from fewer clips. By sacrificing the fine-grained motion within a single video, the model frees up memory to store a much more diverse set of historical instances.
Standard Rehearsal: [Frame 1][Frame 2][Frame 3][Frame 4] -> (Only 1 Video Clip Saved)
SMILE Rehearsal: [Video A Frame][Video B Frame][Video C Frame][Video D Frame] -> (4 Distinct Videos Saved)
Taking this concept a step further, Pei et al. developed FrameMaker. Instead of merely selecting an existing frame from a video, FrameMaker uses a learnable frame-weighting mechanism to condense an entire video clip's information into a single, highly informative synthetic frame. This approach preserves the semantic footprint of the temporal sequence while slashing memory consumption by 80%.
Shifting the Buffer: Feature-Space and Generative Rehearsal
Rather than trying to compress raw pixels, another highly effective strategy shifts the rehearsal buffer away from the input space entirely.
- Feature Buffers: The authors of uVCIL bypass raw video storage by saving low-dimensional embeddings within a feature-based memory buffer managed by deep clustering. Similarly, for real-time streaming of long videos, Zeng et al. introduce a "Persistent Event Memory Forest" that compresses historical visual features, allowing models to maintain a long-term temporal context without experiencing memory exhaustion.
- Generative Rehearsal: If storing data is too expensive, why not generate it on the fly? Chen et al. introduce Continual Predictive Learning (CPL), where a world model "dreams up" past video sequences using a predictive experience replay mechanism. This eliminates the physical storage bottleneck by substituting memory with generative computation.
Exemplar-Free Learning via Parameter Isolation and Prompting
The most radical solution to the video storage overhead is to eliminate the replay buffer entirely. Recent breakthroughs achieve this by isolating parameters or leveraging frozen foundation models.
Frequency-Domain Subnetworks
Instead of allocating new hardware memory for data, models can allocate internal capacity. Kang et al. developed Progressive Fourier Neural Representation (PFNR), which identifies sparse, non-overlapping "winning ticket" subnetworks within the frequency (Fourier) domain. By isolating tasks in the frequency domain, a single network can compile and store multiple videos sequentially without cross-task interference or data rehearsal.
Space-Time Prompting
With the rise of powerful vision-language models like CLIP, researchers have realized that the core visual representations do not need to be updated at all. By freezing the heavy foundational weights, the risk of catastrophic forgetting in the backbone drops to zero.
To adapt these frozen models to sequential video tasks, Villa et al. (PIVOT) and Pei et al. (ST-Prompt) introduce lightweight, learnable prompt blocks. These space-time prompts capture task-specific temporal dynamics without altering the underlying model parameters. Because the prompts are incredibly lightweight, this strategy delivers state-of-the-art incremental performance while utilizing less than 1% of the memory footprint required by traditional video rehearsal methods.
Momentum Review
Alternatively, models can review their own history without external data. Gao et al. introduced BMU-MoCo, which utilizes a bidirectional momentum update strategy. Here, the active encoders directly "review" past knowledge from their slowly moving momentum counterparts, stabilizing the representation space across sequential datasets without requiring a single historical video file.
Key Foundational Work
vCLIMB: A Novel Video Class Incremental Learning Benchmark (2022) — contributed 3 concept elements to this direction.
Class-Incremental Learning for Action Recognition in Videos (2021) — contributed 2 concept elements to this direction.
BMU-MoCo: Bidirectional Momentum Update for Continual Video-Language Modeling (2022) — contributed 1 concept elements to this direction.