Abstract
Class-Incremental Learning (CIL) aims to develop models that can learn new tasks sequentially without forgetting previous knowledge. In the video domain (VCIL), this is complicated by the need to capture both complex spatial and temporal information. Traditional methods often rely on "exemplar rehearsal"—storing a subset of old data—which poses significant privacy and memory challenges. We introduce StPR (Spatiotemporal Preservation and Routing), a novel exemplar-free VCIL framework. StPR utilizes Frame-Shared Semantics Distillation (FSSD) to protect critical spatial features and a Temporal Decomposition-based Mixture-of-Experts (TD-MoE) to dynamically route temporal dynamics. Our approach significantly outperforms existing exemplar-free methods and even rivals rehearsal-based techniques on major action recognition benchmarks like UCF101 and Kinetics-400.
The Challenge of Video Incremental Learning
In the quest to build AI that learns like humans, Class-Incremental Learning (CIL) is a critical frontier. The goal is to pick up new skills over time without overwriting existing ones—a phenomenon known as catastrophic forgetting. When applied to video, the challenge scales significantly because models must recognize both objects (spatial) and motion (temporal).
Most current solutions mitigate forgetting by keeping a small subset of old videos to "remind" the model. However, storing video data raises serious privacy concerns and consumes massive amounts of memory. StPR addresses these issues by providing a high-performing framework that is entirely exemplar-free, disentangling "what" is in a scene from "how" it moves.
The StPR Framework
StPR sits on top of a frozen CLIP (Vision Transformer) backbone and introduces two primary innovations to handle the spatiotemporal nature of video data without needing past exemplars.
1. Frame-Shared Semantics Distillation (FSSD)
Spatial features are often shared across many different actions. FSSD identifies "semantically stable" neural network channels using a combination of Semantic Sensitivity (via Fisher Information) and Classification Contribution scores. By applying selective regularization, the model "locks" these critical channels to preserve old knowledge while allowing less vital channels to adapt to new tasks. This surgical approach balances stability and plasticity more effectively than traditional "blind" distillation.
2. Temporal Decomposition-based Mixture-of-Experts (TD-MoE)
Temporal cues are often highly task-specific. StPR assigns a dedicated "expert" (a small set of trainable parameters) to each new task. To determine which expert to use during inference without a task ID, the model isolates temporal residuals—the subtle differences between frames that represent motion. By comparing these motion patterns to "anchors" of previous tasks, a router dynamically weights the experts' contributions.
Experimental Evaluation
The researchers evaluated StPR on four major action recognition benchmarks: UCF101, HMDB51, Something-Something V2 (SSv2), and Kinetics-400. They utilized the TCD and vCLIMB benchmarks to simulate incremental learning stages.
Key Results
- Superior Accuracy: On the UCF101 10-task split, StPR achieved 94.67% average accuracy, significantly outperforming the previous state-of-the-art exemplar-free method, ST-Prompt (84.75%).
- Efficiency: Despite storing no old data, StPR outperformed many rehearsal-based methods, proving that architectural disentanglement can be more effective than data replay.
- Low Forgetting: Backward Forgetting (BWF) metrics remained notably low, indicating that the FSSD module successfully protected the model's memory of earlier tasks.
Conclusion and Future Work
StPR demonstrates that disentangling spatial semantics from temporal dynamics is a powerful strategy for exemplar-free learning. By treating motion as a unique "fingerprint" for task routing, the model maintains high performance across sequential tasks.
Future work will focus on scaling the number of experts in the MoE module to improve long-term efficiency, exploring open-world settings where models encounter unseen classes, and optimizing the framework for deployment on resource-constrained edge computing devices.
Summary updated