Abstract
Although few-shot action recognition based on the metric learning paradigm has achieved significant success, it fails to address two critical issues: (1) inadequate action relation modeling and underutilization of multi-modal information; and (2) challenges in handling video matching problems with different lengths, speeds, and misalignment of video sub-actions. To address these issues, we propose a Two-Stream Joint Matching method based on contrastive learning (TSJM), which consists of two modules: the Multi-modal Contrastive Learning Module (MCL) and the Joint Matching Module (JMM). The objective of the MCL is to investigate inter-modal mutual information relationships, thereby extracting modal information to enhance action relationship modeling. The JMM aims to simultaneously address video matching problems. The effectiveness of the proposed method is evaluated on two widely used few-shot action recognition datasets, SSv2 and Kinetics. Comprehensive ablation experiments substantiate the efficacy of our approach.
Introduction
Few-shot action recognition (FSAR) aims to recognize new categories of human actions using only a handful of labeled examples. While metric-learning approaches have made strides, they face two persistent hurdles. First, most models rely heavily on RGB frames (appearance), underutilizing motion information like optical flow, which is vital for distinguishing similar actions (e.g., "opening a box" vs. "closing a box"). Second, videos of the same action often vary in speed and duration, and "sub-actions" within a video might be misaligned or even reversed. Standard alignment tools often fail when the complexity of these sub-actions increases.
Methodology: The TSJM Framework
The TSJM framework addresses these challenges through a dual-stream architecture that integrates multi-modal features and employs a sophisticated matching strategy.
Multi-modal Contrastive Learning (MCL)
To effectively use both RGB and Optical Flow without requiring extra labels, the authors utilize contrastive learning. Since RGB and Flow features reside in different data domains, a bottleneck MLP adapter is introduced to project both modalities into a shared feature space. The model is trained using InfoNCE Loss to maximize the mutual information between the RGB and Flow streams of the same video (positive pairs) while minimizing it for different videos (negative pairs). This forces the model to learn a robust, motion-aware representation of the action.
Joint Matching Module (JMM)
Once features are extracted, the model compares a "query" video to a "support" (example) video using a two-pronged matching approach:
- Ordered Temporal Matching (DTW): Dynamic Time Warping aligns videos of different lengths and speeds while preserving the chronological order of frames.
- Bipartite Graph Matching (KM Algorithm): To handle sub-action misalignment, frames are treated as nodes in a bipartite graph. The Kuhn-Munkres (KM) algorithm finds the "maximum weight perfect matching," allowing the model to find the best global correspondence between frames, even if sub-actions occur out of sequence.
The final similarity score is a weighted average of the scores from both matching strategies across both modalities.
Experimental Evaluation
The model was evaluated on two "gold standard" FSAR benchmarks: Kinetics-100, focusing on diverse human actions, and Something-Something v2 (SSv2), where object interaction and motion direction (e.g., "Pushing something from left to right") are critical.
Performance Comparison
TSJM achieves state-of-the-art (SOTA) performance across both benchmarks, particularly excelling in the challenging 1-shot scenarios.
As shown in the table above, TSJM achieved 75.0% accuracy on Kinetics (5-way 1-shot), outperforming recent models like MoLo (74.0%) and MTFAN (74.6%). On the SSv2 dataset, TSJM reached 58.5% accuracy, representing a significant 15.7% improvement over the OTAM baseline and surpassing the previous SOTA (MoLo at 56.6%).
Key Insights and Conclusion
The success of TSJM stems from using contrastive learning as a bridge between modalities. Instead of simple concatenation, the adapter allows the model to distill essential action relationships common to both appearance and motion. Furthermore, the synergy between DTW and Bipartite Graph Matching provides a "global" view that makes the model resilient to variations in how different individuals perform the same sub-tasks.
While the reliance on dense optical flow introduces some preprocessing overhead, the performance gains—especially on motion-heavy datasets like SSv2—demonstrate the value of the TSJM approach. Future work may explore "on-the-fly" motion feature extraction to reduce computational complexity while maintaining the benefits of multi-modal joint matching.
Summary updated