Few-Shot Video Classification via Representation Fusion and Promotion Learning

Haifeng Xia, Kai Li, Martin Renqiang Min, Zhengming Ding

Abstract

Few-shot video classification (FSVC) aims to recognize novel action categories using only a limited number of labeled examples. This task is inherently challenging due to semantic deficiency in sparse samples and temporal redundancy within video sequences. We present the Representation Fusion and Promotion Learning (RFPL) framework, a novel approach that enhances video representations by integrating global meta-action knowledge and employing reinforcement learning for dynamic frame weighting. Our framework serves as a plug-and-play enhancement for existing state-of-the-art models, demonstrating significant performance gains across multiple benchmark datasets.

Introduction

Teaching machines to recognize complex actions from just one or two video clips is a fundamental goal in computer vision. Unlike static image classification, few-shot video classification must contend with the high-dimensional nature of temporal data and the presence of irrelevant "noise" frames. Current methods often struggle with two primary bottlenecks: semantic deficiency, where limited samples fail to capture the full scope of an action, and frame redundancy, where discriminative motions are buried among static or irrelevant backgrounds.

The RFPL framework architecture, featuring Meta-Action Learning (MAL) and Reinforced Image Representation (RIR) modules.

Methodology

The RFPL framework addresses these challenges through two specialized modules designed to refine and promote video features.

Meta-Action Learning (MAL)

To overcome semantic deficiency, the MAL module maintains a global meta-action bank ($D$). This bank acts as a repository of "atomic action snippets" learned across the entire training set. During inference, the model uses Singular Value Decomposition (SVD) to fuse these global priors with local video features. This process enriches the task-specific representation with broader semantic context, allowing the model to understand a new category as a composition of known atomic actions.

Reinforced Image Representation (RIR)

To mitigate frame redundancy, the RIR module utilizes a Deep Q-Network (DQN) agent to perform dynamic frame weighting. The agent evaluates pairs of frames and learns a policy to increase the importance of discriminative frames while demoting redundant ones. The agent is trained with a reward signal based on the similarity between videos of the same class, effectively "promoting" the most informative parts of the sequence for classification.

Experimental Results

We evaluated the RFPL framework on three major benchmarks: UCF101, Kinetics, and SSv2-Full. The framework consistently improved the performance of baseline models such as OTAM, TRX, and STRM.

  • SSv2-Full: On the challenging 5-way 5-shot task, RFPL boosted the OTAM baseline from 52.3% to 61.0%, an 8.7% absolute improvement.
  • Kinetics: Integrated with the STRM model, RFPL achieved a state-of-the-art accuracy of 87.7% in the 5-shot setting.
  • UCF101: Consistent gains of 2% to 5% were observed in 1-shot scenarios across various base architectures.
Visualization of the matching process between a query (Dancing Ballet) and support (Dancing Charleston) using the RFPL-enhanced OTAM model.

Discussion and Conclusion

The RFPL framework demonstrates that few-shot tasks benefit significantly from global knowledge priors and goal-oriented attention mechanisms. By treating actions as compositions of meta-actions and using reinforcement learning to filter temporal noise, RFPL provides a robust and flexible solution for video recognition. While the inclusion of SVD and DQN agents introduces some computational overhead, the consistent performance boosts across diverse datasets and architectures highlight the framework's effectiveness as a general-purpose booster for few-shot video classification.

Summary updated

Abstract

Recent few-shot video classification (FSVC) works achieve promising performance by capturing similarity across support and query samples with different temporal alignment strategies or learning discriminative features via Transformer block within each episode. However, they ignore two important issues: a) It is difficult to capture rich intrinsic action semantics from a limited number of support instances within each task. b) Redundant or irrelevant frames in videos easily weaken the positive influence of discriminative frames. To address these two issues, this paper proposes a novel Representation Fusion and Promotion Learning (RFPL) mechanism with two sub-modules: meta-action learning (MAL) and reinforced image representation (RIR). Concretely, during training stage, we perform online learning for seeking a task-shared meta-action bank to enrich task-specific action representation by injecting global knowledge. Besides, we exploit reinforcement learning to obtain the importance of each frame and refine the representation. This operation maximizes the contribution of discriminative frames to further capture the similarity of support and query samples from the same category. Our RFPL framework is highly flexible that it can be integrated with many existing FSVC methods. Extensive experiments show that RFPL significantly enhances the performance of existing FSVC models when integrated with them.
BibTeX
@inproceedings{xia2023few,
  author = {Haifeng Xia and Kai Li and Martin Renqiang Min and Zhengming Ding},
  title = {{Few-Shot Video Classification via Representation Fusion and Promotion Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01769},
}
Figure