Learning deformable action templates from cluttered videos

Benjamin Z. Yao, Song Chun ZhuICCV 2009 · OpenReview

Abstract

Recognizing human actions in real-world, cluttered videos remains a significant challenge due to background noise and the high cost of manual annotation. This paper introduces the Deformable Action Template (DAT), a generative model learnable from weakly supervised data. A DAT is a sequence of image templates composed of shape (Gabor wavelets) and motion (optical flow) primitives. We propose a three-step semi-supervised learning procedure that uses Dynamic Space-Time Warping (DSTW) to align actions in unlabeled videos across a 5D search space. Our approach achieves superior performance on cluttered datasets like CMU and competitive results on the KTH benchmark.

Introduction

Traditional action recognition models often rely on clean silhouettes or precise tracking, which are rarely available in "in-the-wild" video sequences. Furthermore, frame-by-frame labeling is prohibitively expensive. This work addresses these issues by developing a model that can be initialized from a single labeled example and then automatically refined using a pool of unlabeled videos, even in the presence of significant background clutter.

The Deformable Action Template (DAT) Model

The DAT model represents an action as a temporal sequence of templates. Each template is defined by a set of local primitives:

  • Shape Primitives: Based on Gabor filters, these capture the orientation and location of edges, effectively modeling the actor's pose.
  • Motion Primitives: Based on local optical flow patches, these capture the velocity and direction of moving body parts.
Deformable Action Template (DAT) structure, showing shape and motion primitives aligned with training video frames.

The model is "deformable" because these primitives are allowed small spatial perturbations, enabling the template to match variations in body shape or performance style. By combining both shape and motion, the model becomes robust to different types of noise: motion primitives help isolate actors in crowded scenes, while shape primitives prevent confusion from background motion (e.g., passing cars).

Semi-Supervised Learning and DSTW

Learning the DAT involves an iterative process designed to minimize manual supervision. After initializing a template from one labeled video, the system enters a loop of detection and refinement.

The core of the detection phase is Dynamic Space-Time Warping (DSTW). Unlike standard 1D signal matching, DSTW searches a 5D space—comprising spatial coordinates (x, y), scale, and the temporal dimensions of both the template and the target video. This allows the model to find the best-fitting action instance regardless of its location, size, or speed.

The iterative learning framework (a) and the 5D Dynamic Space-Time Warping algorithm (b) used for action detection and alignment.

Once actions are detected and aligned across the training set, a "shared pursuit" algorithm updates the template by selecting the most consistent primitives. This iterative refinement allows the model to "learn" the most salient features of an action from cluttered data.

Key Results and Insights

The DAT model was evaluated on two major benchmarks:

  1. CMU Action Dataset: In this highly cluttered environment, the DAT model significantly outperformed baselines. For example, in "jumping jack" detection, the Average Area Under the Curve (AUC) reached 0.43, nearly doubling the 0.22 achieved by previous methods.
  2. KTH Dataset: On this standard classification benchmark, the model achieved 87.8% precision, proving that a model designed for clutter remains highly competitive in cleaner environments.

Ablation studies highlighted the synergy between primitives. For the "Push Button" action, using motion alone yielded an AUC of 0.12, which jumped to 0.74 when combined with shape primitives.

Conclusion and Limitations

The Deformable Action Template provides a robust, interpretable framework for action recognition with minimal supervision. While the 5D search space is computationally intensive—often requiring several minutes per clip—the model's ability to handle temporal variations and background clutter marks a significant step forward. Future work may focus on optimizing the search algorithm and extending the model to handle more extreme 3D viewpoint changes.

Summary updated

Abstract

In this paper, we present a Deformable Action Template (DAT) model that is learnable from cluttered real-world videos with weak supervisions. In our generative model, an action template is a sequence of image templates each of which consists of a set of shape and motion primitives (Gabor wavelets and optical-flow patches) at selected orientations and locations. These primitives are allowed to slightly perturb their locations and orientations to account for spatial deformations. We use a shared pursuit algorithm to automatically discover a best set of primitives and weights by maximizing the likelihood over one or more aligned training examples. Since it is extremely hard to accurately label human actions from real-world videos, we use a three-step semi-supervised learning procedure. 1) For each human action class, a template is initialized from a labeled (one bounding-box per frame) training video. 2) The template is used to detect actions from other training videos of the same class by a dynamic space-time warping algorithm, which searches a best match between the template and target video in 5D space (x, y, scale, t <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">template</sub> and t <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">target</sub> ) using dynamic programming. 3) The template is updated by the shared pursuit algorithm over all aligned videos. The 2nd and 3rd steps iterate several times to arrive at an optimal action template. We tested our algorithm on a cluttered action dataset (the CMU dataset) and achieved favorable performance than. Our classification performance on the KTH dataset is also comparable to state-of-the-arts.
BibTeX
@inproceedings{yao2009learning,
  author = {Benjamin Z. Yao and Song Chun Zhu},
  title = {{Learning deformable action templates from cluttered videos}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2009},
  url = {https://openreview.net/forum?id=HkV1PWfdbS},
}
Figure