Rethinking Matching-Based Few-Shot Action Recognition

Juliette Bertrand, Yannis Kalantidis, Giorgos Tolias

Introduction

Few-shot action recognition (FSAR) is a challenging computer vision task that requires models to identify novel action categories using only a handful of labeled examples. Traditionally, the research community has operated under the assumption that because video is a sequential medium, the primary challenge lies in temporal alignment—ensuring that the timing of a "query" video matches the timing of a "support" video. This has led to the development of increasingly complex algorithms, such as Dynamic Time Warping (DTW) and Ordered Temporal Alignment Module (OTAM), designed to synchronize these sequences.

This paper challenges this prevailing wisdom. The authors suggest that the perceived need for complex temporal alignment may be a byproduct of using weak, 2D image-based backbones that fail to capture motion effectively before the matching stage. By introducing a modern spatio-temporal backbone and a simple, non-temporal matching function called Chamfer++, the authors demonstrate that "what" happens in a video is often more discriminative than exactly "when" it happens.

Methodology

The authors propose a shift in focus from complex matching architectures to robust feature representation. Their approach is built on three main pillars:

1. Spatio-Temporal Backbone

Instead of standard 2D ResNet architectures, the authors utilize an R(2+1)D backbone. This architecture treats video clips as 3D volumes, allowing the feature extractor to capture temporal dynamics and motion patterns directly within the feature maps. By the time the data reaches the matching stage, the features already encode significant temporal context.

2. Chamfer Matching and Joint-Matching (Chamfer+)

The core matching logic is based on the symmetric Chamfer distance. For every clip in a query video, the model identifies the most similar clip in the support video (and vice versa), averaging these scores to determine similarity. This method is entirely non-temporal; it does not enforce a specific chronological order.

In multi-shot scenarios (e.g., 5-shot), the authors introduce Joint-Matching (Chamfer+). Rather than comparing the query to each support video individually and averaging the results, they pool all clips from all support videos of a specific class into a single "super-set." The query is then matched against this collective set, providing a more comprehensive representation of the action category.

3. Clip Tuples (Chamfer++)

To further refine the matching process, Chamfer++ utilizes "tuples"—pairs or triplets of clips—rather than single isolated clips. This allows the model to capture higher-order relationships and broader context. Crucially, the matching function remains parameter-free and non-temporal, ignoring the global sequence order of these tuples.

Experimental Evaluation

The proposed method was evaluated on three standard benchmarks: Kinetics-100, UCF-101, and Something-Something V2 (SS-v2). The latter is particularly significant as it is a "temporally-heavy" dataset where the direction of motion (e.g., "moving something left to right") is the primary differentiator between classes.

Key Results

  • State-of-the-Art Performance: Chamfer++ achieved new state-of-the-art (SOTA) results for 1-shot recognition across all three datasets.
  • The Power of the Backbone: The authors found that simply upgrading from a 2D to a 3D backbone yielded an accuracy boost of over 10%, regardless of the matching algorithm used. This suggests that many previous innovations in matching were compensating for weak feature extractors.
  • Success on SS-v2: On the temporally-sensitive SS-v2 dataset, Chamfer++ reached 67.8% accuracy in the 1-shot setting. This significantly outperformed previous complex matching methods like TRX, which scored 42.0% in its original implementation (and 65.5% when re-implemented with the R(2+1)D backbone).

Discussion and Insights

The findings of this paper offer several critical insights for the field of FSAR:

  1. Order is Secondary: Once features are sufficiently strong, non-temporal matching often performs as well as, or better than, complex temporal alignment. This suggests that the "essence" of an action is often captured in its constituent parts rather than its strict linear progression.
  2. Simplicity and Interpretability: Chamfer++ is parameter-free and highly interpretable. It allows researchers to visualize exactly which clips in a query video correspond to clips in the support set, making the model's decision-making process transparent.
  3. Efficiency: Because the matching stage requires no training and avoids complex optimization (like those found in TSL methods), it is computationally efficient and easier to deploy.

Conclusion

By rethinking the necessity of temporal alignment, the authors show that a strong spatio-temporal backbone combined with a simple Chamfer-based matching function can outperform the current state-of-the-art. The success of Chamfer++ suggests that future research in few-shot action recognition should prioritize the learning of better representations over the engineering of increasingly complex matching layers. The study serves as a reminder that in machine learning, a more sophisticated model is not always a better one; often, the key to progress lies in identifying the right level of abstraction for the features themselves.

Summary updated

Abstract

Few-shot action recognition, i.e. recognizing new action classes given only a few examples, benefits from incorporating temporal information. Prior work either encodes such information in the representation itself and learns classifiers at test time, or obtains frame-level features and performs pairwise temporal matching. We first evaluate a number of matching-based approaches using features from spatio-temporal backbones, a comparison missing from the literature, and show that the gap in performance between simple baselines and more complicated methods is significantly reduced. Inspired by this, we propose Chamfer++, a non-temporal matching function that achieves state-of-the-art results in few-shot action recognition. We show that, when starting from temporal features, our parameter-free and interpretable approach can outperform all other matching-based and classifier methods for one-shot action recognition on three common datasets without using temporal information in the matching stage. Project page: https://jbertrand89.github.io/matching-based-fsar
BibTeX
@inproceedings{bertrand2023rethinking,
  author = {Juliette Bertrand and Yannis Kalantidis and Giorgos Tolias},
  title = {{Rethinking Matching-Based Few-Shot Action Recognition}},
  booktitle = {SCIA},
  year = {2023},
  url = {https://openreview.net/forum?id=UVun774A4JO},
}
Figure