Abstract
We present a novel method for few-shot video classification, which performs appearance and temporal alignments. In particular, given a pair of query and support videos, we conduct appearance alignment via frame-level feature matching to achieve the appearance similarity score between the videos, while utilizing temporal order-preserving priors for obtaining the temporal similarity score between the videos. Moreover, we introduce a few-shot video classification framework that leverages the above appearance and temporal similarity scores across multiple steps, namely prototype-based training and testing as well as inductive and transductive prototype refinement. To the best of our knowledge, our work is the first to explore transductive few-shot video classification. Extensive experiments on both Kinetics and Something-Something V2 datasets show that both appearance and temporal alignments are crucial for datasets with temporal order sensitivity such as Something-Something V2. Our approach achieves similar or better results than previous methods on both datasets.
Introduction
Few-shot learning (FSL) aims to classify new categories using only a handful of labeled examples. While FSL is well-studied for static images, videos add a layer of complexity: the temporal dimension. Existing methods often struggle with two extremes: they either ignore temporal order entirely (treating videos like a "bag of frames") or they use rigid, computationally expensive alignment algorithms like Dynamic Time Warping (DTW) or Optimal Transport (OT).
This paper addresses these gaps by proposing a flexible alignment strategy that explicitly balances appearance (what is in the frame) and temporal (the order of events) alignments. Furthermore, it introduces the first-ever transductive approach for video few-shot learning, where the model looks at all unlabeled query videos simultaneously to improve its internal class prototypes.
Proposed Methodology
The proposed framework calculates two distinct similarity scores to compare a query video ($X$) and a support video ($Y$).
1. Appearance Similarity Score
The model calculates a pairwise cosine similarity matrix ($D$) between all frames of both videos. It then uses a "log-sum-exp" operator to find the best matches for each frame, regardless of their order. This captures the visual content of the actions.
2. Temporal Similarity Score
To capture the sequence of events, the authors use a temporal order-preserving prior ($\tilde{T}$). They assume that for two videos of the same class, the temporal progression should roughly follow the diagonal of the similarity matrix. This is modeled as a 2D Gaussian distribution. The temporal similarity is then derived by measuring the Kullback-Leibler (KL) divergence between the normalized appearance matrix ($\tilde{D}$) and this diagonal prior.
Prototype Refinement
The model leverages these scores in two settings:
- Inductive Refinement: Class prototypes are refined using only the provided support samples.
- Transductive Refinement: The model uses a "Soft K-means" approach to update prototypes using the distribution of the unlabeled query videos, effectively "clustering" the new data to find better class representatives.
Experimental Results and Insights
The authors evaluated their method on major benchmarks including Kinetics (order-insensitive actions) and Something-Something V2 (SSv2, order-sensitive actions).
- The Importance of Order: A key finding is that temporal alignment is critical for datasets like SSv2 (e.g., distinguishing "pushing" from "pulling") but provides less benefit for Kinetics. The model uses a hyperparameter ($\beta$) to weight these two scores appropriately.
- Transduction as a Performance Booster: Moving from inductive to transductive inference provided a massive performance boost—up to 9% on Kinetics. By considering the query set as a whole, the model better understands the boundaries between new classes.
- Efficiency: Unlike previous iterative alignment methods, this approach is non-iterative and computationally efficient, maintaining the same parameter count as a standard ResNet-50 baseline.
On Kinetics, the model achieved 93.33% accuracy in the transductive 5-way 5-shot setting. On SSv2, it reached 69.42% accuracy, setting a new state-of-the-art.
Conclusion
This research demonstrates that balancing appearance and temporal alignments is essential for robust few-shot video classification. By introducing transductive prototype refinement and a simple yet effective temporal prior, the proposed method achieves superior performance with high computational efficiency. Future work may involve more flexible, learnable temporal priors to handle complex, non-linear action structures.
Summary updated