Learning to Track for Spatio-Temporal Action Localization

Philippe Weinzaepfel, Zaïd Harchaoui, Cordelia Schmid

Abstract

We propose an effective approach for spatio-temporal action localization in realistic videos. The approach first detects proposals at the frame-level and scores them with a combination of static and motion CNN features. It then tracks high-scoring proposals throughout the video using a tracking-by-detection approach. Our tracker relies simultaneously on instance-level and class-level detectors. The tracks are scored using a spatio-temporal motion histogram, a descriptor at the track level, in combination with the CNN features. Finally, we perform temporal localization of the action using a sliding-window approach at the track level. We present experimental results for spatio-temporal localization on the UCF-Sports, J-HMDB and UCF-101 action localization datasets, where our approach outperforms the state of the art with a margin of 15%, 7% and 12% respectively in mAP.

Introduction

Action recognition research has traditionally focused on classification—identifying an action within a pre-trimmed video clip. However, real-world applications such as autonomous surveillance or automated video editing require localization: determining both the spatial bounding box and the temporal boundaries of an action within long, unedited footage.

The primary challenge in this domain is maintaining temporal consistency. Frame-independent detections often suffer from jitter or disappearance due to motion blur and pose variations. Conversely, simple tracking methods are prone to drift. This research addresses these issues by "learning to track," bridging the gap between detection and robust temporal linking.

Methodology

The proposed framework utilizes a four-stage pipeline designed for robustness and accuracy.

Overview of the proposed spatio-temporal action localization pipeline, from frame-level candidates to temporal detection.
  1. Proposal Extraction and Scoring: The system generates candidate regions in each frame using the EdgeBoxes algorithm. These proposals are scored using a "two-stream" CNN architecture that processes both static appearance (RGB) and motion (optical flow) features.
  2. Tracking-by-Detection: Rather than simple frame-to-frame linking, the system employs a tracking-by-detection strategy. For high-scoring proposals, it trains an instance-level detector (specific to the object in that video) and combines it with a class-level detector (e.g., a general "diver" model). This dual-detector approach ensures the tracker remains resilient to significant changes in pose or lighting.
  3. Track-Level Scoring: Generated tracks are evaluated using a novel Spatio-Temporal Motion Histogram (STMH). This descriptor captures action dynamics across multiple frames, aggregating motion information at the track level to distinguish between visually similar actions like walking and running.
  4. Temporal Localization: To identify the precise start and end of an action within a track, a multi-scale sliding window approach is applied at the track level.

Experimental Results

The approach was evaluated on three major benchmarks: UCF-Sports, J-HMDB, and a subset of UCF-101. The method demonstrated significant performance gains, setting new state-of-the-art benchmarks across all datasets.

Qualitative results showing successful spatio-temporal localization for diving and weightlifting actions.
  • UCF-Sports: Achieved 90.5% mAP, representing a 15% increase over previous methods.
  • J-HMDB: Reached 60.7% mAP, a 7% improvement.
  • UCF-101: Achieved 46.77% mAP, outperforming the previous best by 12%.

The tracking approach proved particularly effective for spatial localization, maintaining stable performance even under strict Intersection over Union (IoU) threshold requirements.

Conclusion

This work demonstrates that combining instance-level tracking with class-level semantic information significantly improves action localization. While the multi-stage pipeline is computationally intensive, the gains in mAP across diverse datasets highlight the importance of temporal consistency. Future work will focus on refining temporal localization in even more complex, untrimmed video environments.

Summary updated

Abstract

We propose an effective approach for spatio-temporal action localization in realistic videos. The approach first detects proposals at the frame-level and scores them with a combination of static and motion CNN features. It then tracks high-scoring proposals throughout the video using a tracking-by-detection approach. Our tracker relies simultaneously on instance-level and class-level detectors. The tracks are scored using a spatio-temporal motion histogram, a descriptor at the track level, in combination with the CNN features. Finally, we perform temporal localization of the action using a sliding-window approach at the track level. We present experimental results for spatio-temporal localization on the UCF-Sports, J-HMDB and UCF-101 action localization datasets, where our approach outperforms the state of the art with a margin of 15%, 7% and 12% respectively in mAP.
BibTeX
@inproceedings{weinzaepfel2015learning,
  author = {Philippe Weinzaepfel and Zaïd Harchaoui and Cordelia Schmid},
  title = {{Learning to Track for Spatio-Temporal Action Localization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=HJEqo-f_ZS},
}
Figure