Learning a discriminative hidden part model for human action recognition

Yang Wang, Greg MoriNeurIPS 2008 · OpenReview

Overview

This paper introduces a discriminative part-based model for human action recognition that combines the "big picture" context of global templates with the flexibility of local part-based features. By extending the Hidden Conditional Random Field (hCRF) framework to include a global "root model," the authors achieve state-of-the-art performance on classic benchmarks, proving that actions are best recognized when observing both the overall structure and localized movements.

The Problem: Global Templates vs. Local Patches

Historically, computer vision researchers were divided on how to represent human actions in video. One camp favored global templates—treating the whole frame or a 3D "space-time shape" as a single unit. This captured the overall motion well but struggled with variations in pose, scale, or occlusion. The other camp favored local patches (similar to "bag-of-words" models), which were flexible but often ignored the spatial relationships between body parts (e.g., a hand is usually near an arm).

The authors identify a specific gap: while part-based models (like hCRFs) were revolutionizing object recognition in static images, they hadn't yet been effectively adapted for video-based action recognition. Local patches in video are often too noisy or ambiguous to stand alone; they require the global context of the entire human figure to be interpreted correctly.

Methods: An Augmented hCRF

The researchers developed an augmented Hidden Conditional Random Field (hCRF) to bridge this gap. Their technical approach breaks down into several key components:

  1. Motion Features: The system starts by computing optical flow between frames. This flow is split into four non-negative channels (representing horizontal and vertical movements in both directions), blurred with a Gaussian kernel, and normalized. This creates a robust "motion descriptor" that handles noise and slight camera jitters.
  2. The Root Model: This is the paper's primary innovation. Before looking at individual parts, the model trains a global "root filter" that analyzes the entire frame. This filter serves two purposes: it provides a global "vote" on the action class, and it automatically initializes the locations of the "parts" by finding the most discriminative regions in the motion field.
Visualization of motion features and root filters. (a) Original frame, (b) optical flow, and (c-e) various learned feature maps and root filters.
  1. Hidden Parts: The model assumes there are "hidden" parts (like a moving leg or a waving hand) that aren't explicitly labeled in the training data. The hCRF treats these part labels as latent variables.
  2. The Potential Functions: The model's "energy" is defined by four components:
  3. Unary Potential: How well does a local patch match a specific part label?
  4. Class-Part Compatibility: How likely is a specific part to appear in a specific action (e.g., a "waving" part in a "waving" action)?
  5. Pairwise Potential: The spatial constraints between parts, modeled as a tree structure.
  6. Root Potential: The global compatibility between the whole frame's motion and the action class.

Key Insights

  • Global + Local = Success: The core insight is that local patches in action recognition are often insufficient on their own. A patch of "upward motion" could be a jump, a wave, or a person standing up. By adding the Root Model, the system gains the context needed to disambiguate these local motions.
  • Automatic Part Discovery: Unlike previous models that required researchers to manually define where "parts" were located, this model uses the root filter's energy to automatically select the most salient patches.
  • Discriminative Training: By using an hCRF, the model doesn't just learn what an action like "walking" looks like; it learns the specific features that distinguish walking from running or jogging.

Datasets and Results

The model was evaluated on two seminal action recognition benchmarks: the Weizmann Dataset (83 video sequences, 9 actions) and the KTH Human Motion Dataset (a larger dataset featuring 6 action types across varying scenarios and scales).

The authors compared their hybrid model against a "global-only" baseline and a "local-only" hCRF:

  • Superiority of the Hybrid: On the Weizmann dataset, the hybrid model achieved 90.3% per-frame accuracy, significantly higher than the global-only (74.7%) or local-only (66.6%) versions.
  • Video-Level Accuracy: When aggregating frame-level predictions across a whole video via majority voting, the accuracy jumped to 97.2% on Weizmann and 87.6% on KTH.
  • Benchmark Performance: These results were comparable to or better than the state-of-the-art at the time of publication, particularly outperforming other part-based and bag-of-words approaches.

Limitations and Future Work

While highly effective, the model has a few limitations. First, it primarily operates on a frame-by-frame basis. While it uses optical flow to capture motion between two frames, it doesn't explicitly model long-term temporal dependencies (e.g., the sequence of phases in a stride). Incorporating more complex temporal structures is a natural next step.

Additionally, the system relies on a "stabilized" video sequence where the person is centered. In "in-the-wild" footage with moving cameras or multiple people, a much more robust tracking and detection preprocessing stage would be required. Finally, the spatial relationship between parts is modeled as a simple tree to keep belief propagation efficient; more complex, loopy graphs might represent the human body better, albeit at a higher computational cost.

Summary updated

Abstract

We present a discriminative part-based approach for human action recognition from video sequences using motion features. Our model is based on the recently proposed hidden conditional random field~(hCRF) for object recognition. Similar to hCRF for object recognition, we model a human action by a flexible constellation of parts conditioned on image observations. Different from object recognition, our model combines both large-scale global features and local patch features to distinguish various actions. Our experimental results show that our model is comparable to other state-of-the-art approaches in action recognition. In particular, our experimental results demonstrate that combining large-scale global features and local patch features performs significantly better than directly applying hCRF on local patches alone.
BibTeX
@inproceedings{wang2008learning,
  author = {Yang Wang and Greg Mori},
  title = {{Learning a discriminative hidden part model for human action recognition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2008},
  url = {https://openreview.net/forum?id=Bk4ecdW_-S},
}
Figure