Action Recognition with Improved Trajectories

Heng Wang, Cordelia SchmidICCV 2013 · OpenReview

Abstract

Recently dense trajectories were shown to be an efficient video representation for action recognition and achieved state-of-the-art results on a variety of datasets. This paper improves their performance by taking into account camera motion to correct them. To estimate camera motion, we match feature points between frames using SURF descriptors and dense optical flow, which are shown to be complementary. These matches are, then, used to robustly estimate a homography with RANSAC. Human motion is in general different from camera motion and generates inconsistent matches. To improve the estimation, a human detector is employed to remove these matches. Given the estimated camera motion, we remove trajectories consistent with it. We also use this estimation to cancel out camera motion from the optical flow. This significantly improves motion-based descriptors, such as HOF and MBH. Experimental results on four challenging action datasets (i.e., Hollywood2, HMDB51, Olympic Sports and UCF50) significantly outperform the current state of the art.

Introduction

Action recognition in real-world video environments, such as movies or user-generated content, faces a significant hurdle: camera motion. When a camera pans, tilts, or zooms, it introduces global motion across the entire frame. For traditional action recognition algorithms, this creates substantial noise, as the resulting trajectories and optical flow vectors reflect the camera's movement rather than the actual human action. While some descriptors like Motion Boundary Histograms (MBH) offer partial robustness to constant motion, they often fail during complex camera maneuvers. This research proposes "Improved Trajectories" (iDT) to stabilize video representations by explicitly separating and canceling global camera motion.

Methodology

The proposed pipeline for generating Improved Trajectories involves several critical steps designed to isolate foreground action from background noise.

Camera Motion Estimation and Compensation

The core of the method assumes that two consecutive frames are related by a homography. To estimate this transformation, the authors match points between frames using a combination of SURF descriptors and dense optical flow. SURF provides robustness to image blur, while dense optical flow ensures a high volume of matches. Once matches are established, RANSAC is used to robustly estimate the homography.

Separating camera-induced background trajectories (white) from foreground human motion (green).

Human Masking and Trajectory Pruning

A significant innovation is the integration of a human detector. Because human movement is often inconsistent with the background, it can bias camera motion estimation if the person occupies a large part of the frame. By masking out detected human regions, the algorithm focuses exclusively on the background. After estimating the camera motion, the second frame is "warped" to align with the first. Trajectories that remain static relative to this stabilized background are pruned, as they likely represent irrelevant background objects.

Experimental Evaluation

The authors evaluated the iDT framework on four challenging "in-the-wild" datasets: Hollywood2, HMDB51, Olympic Sports, and UCF50. The results demonstrated a substantial leap over previous state-of-the-art (SOTA) methods.

  • HMDB51: Reached 57.2% accuracy, a significant jump from the previous best of ~52%.
  • Hollywood2: Improved from 62.5% to 64.3% (mAP).
  • Olympic Sports: Jumped from 83.2% to 91.1% (mAP).
  • UCF50: Improved from 83.3% to 91.2% accuracy.

The performance gains were particularly notable in HOF (Histogram of Optical Flow) descriptors, which previously suffered heavily from camera noise but became highly competitive after stabilization.

Key Insights and Discussion

The success of Improved Trajectories highlights several key principles in video analysis. First, explicit stabilization is crucial; even "motion-robust" descriptors like MBH benefit significantly from camera motion compensation. Second, human detection is a critical component for robust estimation in scenes where humans dominate the frame.

Application of improved trajectories to a movie scene, illustrating the effectiveness of motion compensation in dynamic environments.

However, the method does have limitations. Severe motion blur can cause feature matching to fail, and complex scene geometry (e.g., significant parallax) may not be perfectly modeled by a single homography. Additionally, the computational cost of human detection and warping adds overhead compared to simpler dense trajectory methods. Despite these challenges, iDT sets a new benchmark for robust action recognition in unconstrained video.

Summary updated

Abstract

Recently dense trajectories were shown to be an efficient video representation for action recognition and achieved state-of-the-art results on a variety of datasets. This paper improves their performance by taking into account camera motion to correct them. To estimate camera motion, we match feature points between frames using SURF descriptors and dense optical flow, which are shown to be complementary. These matches are, then, used to robustly estimate a homography with RANSAC. Human motion is in general different from camera motion and generates inconsistent matches. To improve the estimation, a human detector is employed to remove these matches. Given the estimated camera motion, we remove trajectories consistent with it. We also use this estimation to cancel out camera motion from the optical flow. This significantly improves motion-based descriptors, such as HOF and MBH. Experimental results on four challenging action datasets (i.e., Hollywood2, HMDB51, Olympic Sports and UCF50) significantly outperform the current state of the art.
BibTeX
@inproceedings{wang2013action,
  author = {Heng Wang and Cordelia Schmid},
  title = {{Action Recognition with Improved Trajectories}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  url = {https://openreview.net/forum?id=rJW2jlfu-r},
}
Figure