Human action recognition from a single clip per action

Weilong Yang, Yang Wang, Greg MoriWorkshops @ ICCV 2009 · OpenReview

Abstract

Learning-based approaches for human action recognition often rely on large training sets. Most of these approaches do not perform well when only a few training samples are available. In this paper, we consider the problem of human action recognition from a single clip per action. Each clip contains at most 25 frames. Using a patch based motion descriptor and matching scheme, we can achieve promising results on three different action datasets with a single clip as the template. Our results are comparable to previously published results using much larger training sets. We also present a method for learning a transferable distance function for these patches. The transferable distance function learning extracts generic knowledge of patch weighting from previous training sets, and can be applied to videos of new actions without further learning. Our experimental results show that the transferable distance function learning not only improves the recognition accuracy of the single clip action recognition, but also significantly enhances the efficiency of the matching scheme.

Introduction

Most state-of-the-art human action recognition models are data-hungry, requiring hundreds or thousands of labeled examples to achieve high accuracy. However, in real-world scenarios like surveillance or specialized video retrieval, massive datasets are often unavailable. This research addresses the "impoverished training data" problem: How can a computer recognize an action (like boxing or clapping) after seeing it only once? This mimics the human ability to generalize from a single example, a feat that traditional machine learning struggles to replicate without overfitting.

Methodology

The proposed framework consists of a robust matching scheme and a transfer learning layer designed to handle single-exemplar templates.

Patch-Based Matching

Instead of analyzing the whole frame, the system decomposes the video into small motion patches. It uses a figure-centric representation (tracking the person) and describes motion using blurred optical flow channels. To compare a "query" video to a "template" video, it performs a local search to find the best-matching patches, accounting for variations in how different people move, such as varying stride lengths.

The patch-based matching scheme identifies corresponding motion regions between query and template clips.

Transferable Distance Function

The core innovation is the learning of a weight matrix (P) from a "source" dataset of known actions. This matrix maps "hyper-features" (visual words representing patch appearance and location) to a saliency weight. The learning is framed as a max-margin optimization problem, ensuring that the distance between similar actions is smaller than the distance between dissimilar actions. This knowledge is then transferred to a "target" dataset with completely different action categories.

Key Insights and Results

The study demonstrates that "saliency" is a transferable property. The model essentially learns that "outstretched limbs" are generally more informative than a "still torso," regardless of the specific action being performed.

Visualization of salient patches (highlighted in red/blue) shows the model prioritizing informative limb movements over the torso.

Performance Benchmarks

The method was evaluated on three prominent benchmarks: the KTH Dataset, the Weizmann Dataset, and a Cluttered Human Action Dataset.

  • One-Shot Performance: On the KTH dataset, using just one clip per action, the model achieved 75.7% accuracy, rivaling early methods that used thousands of samples.
  • Superiority over Baselines: On the Weizmann dataset, the "Direct Comparison" method achieved 87% accuracy, outperforming other metric learning approaches under the same single-exemplar constraints.
  • Efficiency via Sparsity: By learning which patches are important, the system doesn't need to match every single patch. Using only the top 5 patches per frame on the cluttered dataset achieved 86.67% accuracy, representing an 87.5% speedup in matching time with only a 3% drop in accuracy.

Conclusion

This approach proves that knowledge of "what to look at" is generic and transferable across action categories. While the method relies on reliable human tracking as a preprocessing step, it significantly reduces the data requirements for action recognition. Future work may explore how this approach scales to more complex, multi-stage activities found in unconstrained, real-world video environments.

Summary updated

Abstract

Learning-based approaches for human action recognition often rely on large training sets. Most of these approaches do not perform well when only a few training samples are available. In this paper, we consider the problem of human action recognition from a single clip per action. Each clip contains at most 25 frames. Using a patch based motion descriptor and matching scheme, we can achieve promising results on three different action datasets with a single clip as the template. Our results are comparable to previously published results using much larger training sets. We also present a method for learning a transferable distance function for these patches. The transferable distance function learning extracts generic knowledge of patch weighting from previous training sets, and can be applied to videos of new actions without further learning. Our experimental results show that the transferable distance function learning not only improves the recognition accuracy of the single clip action recognition, but also significantly enhances the efficiency of the matching scheme.
BibTeX
@inproceedings{yang2009human,
  author = {Weilong Yang and Yang Wang and Greg Mori},
  title = {{Human action recognition from a single clip per action}},
  booktitle = {Workshops @ ICCV},
  year = {2009},
  url = {https://openreview.net/forum?id=t5zbSgZhhVy},
}
Figure