Space-time Prompting for Video Class-incremental Learning

Yixuan Pei, Zhiwu Qing, Shiwei Zhang, Xiang Wang, Yingya Zhang, Deli Zhao, Xueming QianICCV 2023 · DOI

Introduction

In the field of machine learning, "Catastrophic Forgetting" remains a significant obstacle for models trained on sequential tasks. When a model learns a new set of categories—such as transitioning from recognizing "walking" to "running"—it often loses the ability to identify previously learned actions. This challenge is particularly acute in Video Class-incremental Learning (VCIL), where the temporal dimension of video data adds complexity beyond static image recognition.

Historically, VCIL solutions have relied on "rehearsal buffers," which store a subset of previous video data to periodically refresh the model's memory. However, storing raw video is memory-intensive and presents substantial data privacy concerns. This paper introduces ST-Prompt (Space-Time Prompting), a rehearsal-free framework that leverages the pre-trained capabilities of vision-language models to achieve state-of-the-art performance with minimal memory overhead.

Methodology

ST-Prompt is built upon a frozen, pre-trained CLIP (Contrastive Language-Image Pre-training) backbone. To prevent forgetting, the model's core weights remain fixed. Instead, the researchers introduce a dual-prompting system composed of learnable vectors that guide the model's inference for new tasks.

1. Task-specific Prompts

These prompts are designed to identify the specific task or class group associated with an input. To capture the multi-faceted nature of video, these prompts are multi-grained:

  • Spatial Prompts: Focus on object and scene recognition within individual frames.
  • Temporal Prompts: Capture the dynamics of how subjects move over time.
  • Comprehensive Prompts: Provide a global context of the action.

A K-Means clustering approach is utilized to match input features to the most relevant prompt in a stored "prompt pool," ensuring the model applies the correct learned knowledge to the current input.

2. Task-agnostic Prompts

Shared across all tasks, these prompts are integrated to provide the frozen image-based CLIP model with "temporal perception." They function as a communication layer, allowing context to be exchanged between different frames in a video sequence. This effectively transforms a static image model into one capable of understanding temporal progression.

Experimental Evaluation

The ST-Prompt framework was evaluated on three primary video action recognition benchmarks: UCF101, HMDB51, and Something-Something V2 (SSv2). The researchers tested various incremental settings, including 25-stage (2 classes per stage) and 5-stage (10 classes per stage) configurations.

Key Results and Memory Efficiency

ST-Prompt demonstrated significant improvements over existing state-of-the-art methods. On the HMDB51 dataset, the model achieved a 9.06% accuracy boost over previous top-performing methods in a 25-stage setting.

Comparison of ST-Prompt against traditional rehearsal-based methods in terms of memory usage and accuracy.

As shown in Figure 1, the most notable advantage of ST-Prompt is its extreme memory efficiency. While traditional rehearsal-based methods (e.g., Stored Videos or Stored Frames) require approximately 585.70 MB of storage for exemplars, ST-Prompt achieves higher accuracy (55.13%) while requiring only 2.98 MB for its prompt pool—a reduction in memory consumption of over 99%.

Discussion and Future Work

The success of ST-Prompt suggests that the knowledge required to mitigate catastrophic forgetting can be effectively compressed into a small set of learnable parameters rather than requiring the storage of raw data. The model showed particularly strong performance on the motion-sensitive SSv2 dataset, validating the effectiveness of the task-agnostic temporal modeling.

However, the authors note certain limitations. The performance is currently dependent on the quality of the pre-trained backbone (CLIP). Furthermore, the current implementation prepends prompts to input embeddings; future research may explore "prefixing" prompts into deeper transformer layers to further enhance performance. Additionally, perfectly identifying task boundaries during inference without external cues remains an ongoing challenge in the field.

Summary updated

BibTeX
@inproceedings{pei2023space,
  author = {Yixuan Pei and Zhiwu Qing and Shiwei Zhang and Xiang Wang and Yingya Zhang and Deli Zhao and Xueming Qian},
  title = {{Space-time Prompting for Video Class-incremental Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01096},
}
Figure