Abstract
Few-shot video classification (FSVC) aims to recognize novel action categories using only a limited number of labeled examples. This task is inherently challenging due to semantic deficiency in sparse samples and temporal redundancy within video sequences. We present the Representation Fusion and Promotion Learning (RFPL) framework, a novel approach that enhances video representations by integrating global meta-action knowledge and employing reinforcement learning for dynamic frame weighting. Our framework serves as a plug-and-play enhancement for existing state-of-the-art models, demonstrating significant performance gains across multiple benchmark datasets.
Introduction
Teaching machines to recognize complex actions from just one or two video clips is a fundamental goal in computer vision. Unlike static image classification, few-shot video classification must contend with the high-dimensional nature of temporal data and the presence of irrelevant "noise" frames. Current methods often struggle with two primary bottlenecks: semantic deficiency, where limited samples fail to capture the full scope of an action, and frame redundancy, where discriminative motions are buried among static or irrelevant backgrounds.
Methodology
The RFPL framework addresses these challenges through two specialized modules designed to refine and promote video features.
Meta-Action Learning (MAL)
To overcome semantic deficiency, the MAL module maintains a global meta-action bank ($D$). This bank acts as a repository of "atomic action snippets" learned across the entire training set. During inference, the model uses Singular Value Decomposition (SVD) to fuse these global priors with local video features. This process enriches the task-specific representation with broader semantic context, allowing the model to understand a new category as a composition of known atomic actions.
Reinforced Image Representation (RIR)
To mitigate frame redundancy, the RIR module utilizes a Deep Q-Network (DQN) agent to perform dynamic frame weighting. The agent evaluates pairs of frames and learns a policy to increase the importance of discriminative frames while demoting redundant ones. The agent is trained with a reward signal based on the similarity between videos of the same class, effectively "promoting" the most informative parts of the sequence for classification.
Experimental Results
We evaluated the RFPL framework on three major benchmarks: UCF101, Kinetics, and SSv2-Full. The framework consistently improved the performance of baseline models such as OTAM, TRX, and STRM.
- SSv2-Full: On the challenging 5-way 5-shot task, RFPL boosted the OTAM baseline from 52.3% to 61.0%, an 8.7% absolute improvement.
- Kinetics: Integrated with the STRM model, RFPL achieved a state-of-the-art accuracy of 87.7% in the 5-shot setting.
- UCF101: Consistent gains of 2% to 5% were observed in 1-shot scenarios across various base architectures.
Discussion and Conclusion
The RFPL framework demonstrates that few-shot tasks benefit significantly from global knowledge priors and goal-oriented attention mechanisms. By treating actions as compositions of meta-actions and using reinforcement learning to filter temporal noise, RFPL provides a robust and flexible solution for video recognition. While the inclusion of SVD and DQN agents introduces some computational overhead, the consistent performance boosts across diverse datasets and architectures highlight the framework's effectiveness as a general-purpose booster for few-shot video classification.
Summary updated