A Baseline on Continual Learning Methods for Video Action Recognition

Giulia Castagnolo, Concetto Spampinato, Francesco Rundo, Daniela Giordano, Simone Palazzo

Abstract

Continual learning (CL) aims to enable artificial intelligence models to learn from a stream of data without suffering from catastrophic forgetting. While CL has been extensively studied in the context of static image classification, its application to Video Action Recognition (VAR) remains relatively unexplored. This paper establishes a comprehensive benchmark for CL in VAR, evaluating ten established methods. Furthermore, the authors propose two memory-efficient modifications—Confidence-Driven Rehearsal (CDR) and Information-Driven Downsampling (IDD)—designed to address the unique storage challenges posed by high-dimensional video data. Our findings indicate that rehearsal-based methods, particularly DER++, define the current state-of-the-art, while strategic sample selection can significantly reduce memory overhead without compromising accuracy.

Introduction

Most modern deep learning models are trained on stationary datasets where all training samples are available simultaneously. However, real-world applications often require models to learn incrementally from a continuous stream of data. The primary obstacle in this paradigm is catastrophic forgetting, where the acquisition of new knowledge (e.g., recognizing a "jump") leads to the overwriting of weights associated with previously learned tasks (e.g., "running").

Video Action Recognition (VAR) presents unique challenges for continual learning. Unlike static images, videos are spatio-temporal and high-dimensional, making them exceptionally memory-intensive. Traditional CL techniques often rely on a "rehearsal buffer"—a small subset of past data kept to refresh the model's memory. In the context of video, storing even a modest buffer can become prohibitively expensive. This research investigates how standard CL methods translate to the video domain and proposes strategies to optimize memory usage.

Methodology

Benchmarking Established Methods

The researchers evaluated ten established CL methods using a consistent R(2+1)D backbone. These methods were categorized into three main approaches:

  1. Regularization-based: Methods like Elastic Weight Consolidation (EWC) that penalize changes to important weights.
  2. Knowledge Distillation: Methods like Learning without Forgetting (LwF) that use the previous model's outputs to guide the current model.
  3. Rehearsal-based: Methods like Experience Replay (ER) and Dark Experience Replay (DER++) that maintain a physical buffer of past samples.

Proposed Memory-Efficient Variants

To address the storage bottleneck of video data, the authors introduced two method-agnostic modifications:

  • Confidence-Driven Rehearsal (CDR): Rather than selecting buffer samples at random, CDR utilizes a prediction threshold ($\delta$) to save only the samples the model identifies with high confidence. By focusing on these "prototypical" samples, the model anchors its knowledge on the most discriminative parts of the data distribution, which is more effective in high-dimensional spaces than attempting to represent the entire distribution with limited samples.
  • Information-Driven Downsampling (IDD): Standard VAR models require fixed-length inputs, often achieved through random frame sampling. IDD instead utilizes the norm of optical flow to identify frames with the highest motion content. By prioritizing these information-dense frames for the buffer, the model retains the most meaningful temporal features of the action.

Experimental Setup

The study utilized a subset of the UCF101 dataset, focusing on 30 distinguishable action classes. These classes were organized into 15 distinct tasks (2 classes per task). The models were evaluated under two scenarios:

  1. Task-Incremental Learning: The model is provided with a task ID at inference time, narrowing the search space.
  2. Class-Incremental Learning: A more challenging scenario where the model must identify the correct class from all previously seen categories without any task-specific hints.

Key Insights and Results

The Dominance of Rehearsal

The benchmark results clearly demonstrate that rehearsal-based methods significantly outperform regularization and distillation approaches. DER++ emerged as the top performer, achieving approximately 47.8% accuracy in the difficult class-incremental setting and over 90% in task-incremental learning. However, this performance comes at a cost: a standard video buffer can add nearly a gigabyte of RAM overhead.

Efficiency Gains via CDR

Confidence-Driven Rehearsal proved to be a powerful tool for memory optimization. By using CDR, a buffer of only 100 samples was able to achieve performance levels comparable to a standard 200-sample buffer. This suggests that the quality and "certainty" of stored samples are more critical than sheer quantity in the video domain.

The Role of Motion and Thresholding

The researchers found that using optical flow (IDD) for frame selection is superior to random sampling, particularly when the total number of stored videos is very low. Additionally, they identified a "sweet spot" for the confidence threshold ($\delta$). While filtering for high confidence helps, setting the threshold too high ($\delta > 0.8$) causes the model's view of the data to become too narrow, leading to a loss of generalization and a subsequent drop in performance.

Limitations and Future Directions

While this study establishes a vital baseline, several areas remain for future exploration:

  • Adaptive Thresholding: The current confidence threshold ($\delta$) must be set manually. Developing an adaptive mechanism that adjusts the threshold based on task difficulty or data diversity could improve robustness.
  • Mitigating Class Bias: Focusing exclusively on high-confidence samples may lead the model to ignore rare variations of an action, potentially introducing biases toward "standard" executions of a movement.
  • Attention-Based Selection: Future iterations could replace optical flow with internal model attention maps to decide which frames or spatial regions are most worthy of preservation in the memory buffer.

Conclusion

This research provides a much-needed framework for applying continual learning to video action recognition. By demonstrating that rehearsal-based methods are effective but memory-heavy, and proving that confidence-driven selection can mitigate these costs, the authors pave the way for more efficient and scalable video-based AI systems.

Summary updated

Abstract

Continual learning has recently attracted attention from the research community, as it aims to solve long-standing limitations of classic supervised-trained models. However, most research on this subject has tackled continual learning in simple image classification scenarios. In this paper, we present a benchmark of state-of-the-art continual learning methods on video action recognition. Besides the increased complexity due to the temporal dimension, the video setting imposes stronger requirements on computing resources for top-performing rehearsal methods. To counteract the increased memory requirements, we present two method-agnostic variants for rehearsal methods, exploiting measures of either model confidence or data information to select memorable samples. Our experiments show that, as expected from the literature, rehearsal methods outperform other approaches; moreover, the proposed memory-efficient variants are shown to be effective at retaining a certain level of performance with a smaller buffer size.
BibTeX
@inproceedings{castagnolo2023baseline,
  author = {Giulia Castagnolo and Concetto Spampinato and Francesco Rundo and Daniela Giordano and Simone Palazzo},
  title = {{A Baseline on Continual Learning Methods for Video Action Recognition}},
  booktitle = {ICIP},
  year = {2023},
  url = {https://openreview.net/forum?id=nwyWdOydiX},
}
Figure