Abstract
The goal of few-shot video classification is to learn a classification model with good generalization ability when trained with only a few labeled videos. However, it is difficult to learn discriminative feature representations for videos in such a setting. In this paper, we propose Temporal Alignment Prediction (TAP) based on sequence similarity learning for few-shot video classification. In order to obtain the similarity of a pair of videos, we predict the alignment scores between all pairs of temporal positions in the two videos with the temporal alignment prediction function. Besides, the inputs to this function are also equipped with the context information in the temporal domain. We evaluate TAP on two video classification benchmarks including Kinetics and Something-Something V2. The experimental results verify the effectiveness of TAP and show its superiority over state-of-the-art methods.
Introduction
Recognizing actions in videos typically requires extensive datasets with hundreds of labeled examples per category. Few-shot video classification addresses this dependency by training models to recognize new actions from just one or a handful of examples. The core challenge lies in establishing a robust similarity metric between a "query" video and a "support" (labeled) video. While simple methods like averaging frame features lose the temporal "story," more advanced techniques using Dynamic Time Warping (DTW) suffer from strict chronological ordering assumptions and high computational costs due to iterative dynamic programming.
The Temporal Alignment Prediction (TAP) Framework
To overcome these limitations, we propose the Temporal Alignment Prediction (TAP) module. TAP replaces rigid algorithms with a flexible, neural-network-based alignment mechanism that captures long-range temporal context and is significantly faster to train.
The TAP process consists of four main stages:
- Feature Extraction: A fixed number of frames (e.g., $T=6$) are sampled and processed through a ResNet-50 backbone to obtain initial features.
- Temporal Message Passing: A Bidirectional LSTM (BiLSTM) provides each frame with context from preceding and succeeding frames, allowing the model to distinguish between different stages of an action (e.g., beginning vs. end).
- Alignment Prediction ($P$): A neural network ($h_\beta$) evaluates every possible pair of frames between two videos to predict an alignment score, indicating the likelihood that the frames represent the same action stage.
- Similarity Calculation ($S$): A similarity matrix is calculated via inner products of features. The final video-to-video similarity score is the sum of this matrix weighted by the predicted alignment scores.
Unlike DTW, TAP does not enforce strict monotonicity, allowing it to handle variations in sub-action order or pacing effectively.
Experimental Evaluation
TAP was evaluated on two major benchmarks: Kinetics, featuring diverse human actions, and Something-Something V2, which focuses on physical interactions where temporal order is critical to distinguishing between opposite actions (e.g., "pushing left to right" vs. "right to left").
Key Results
- Performance: TAP achieves state-of-the-art results on both benchmarks, outperforming established methods like TAM and ARN.
- Efficiency: By predicting alignment scores simultaneously rather than through iterative path-finding, TAP is approximately 2x faster to train than DTW-based approaches. On a 4-GPU setup, TAP required 5 hours to train compared to 10 hours for TAM.
- Ablation Success: Comparisons with a "FewShot-TSN" baseline (using simple average pooling) demonstrate that explicit temporal alignment modeling is essential for high-performance video classification.
Conclusion
TAP demonstrates that flexible, context-aware alignment prediction is superior to rigid algorithmic constraints for few-shot video tasks. By leveraging parallel computation, it offers a significant speed advantage. Future research will explore higher sampling densities to capture more nuanced actions and evaluate the model's performance on longer, multi-event video sequences.
Summary updated