Progression-Guided Temporal Action Detection in Videos

Chongkai Lu, Man‐Wai Mak, Ruimin Li, Zheru Chi, Hong Fu

TL;DR

The Action Progression Network (APN) redefines temporal action detection by treating actions as a continuous evolution of 101 ordered stages (0% to 100%) rather than discrete categories. By training end-to-end to recognize these "action progressions," the framework achieves state-of-the-art precision, particularly for long-duration actions, and offers exceptionally fast inference speeds.

What Problem Does This Paper Address?

Temporal Action Detection (TAD) is the "needle in a haystack" problem of computer vision: given a long, untrimmed video, a model must identify exactly when an action starts, when it ends, and what category it belongs to.

Current "bottom-up" methods typically classify video snippets into three broad stages: starting, course, and ending. The authors argue this is too coarse. It ignores the internal chronological structure of an action—for example, a "high jump" looks very different at 20% completion (running) versus 80% (clearing the bar). Furthermore, many existing models rely on pre-extracted offline features, which prevents the entire system from being fine-tuned end-to-end and often makes them sensitive to the specific "background style" of the training videos.

Methods

The core innovation is the Action Progression Network (APN). Instead of simple classification, the authors quantify an action into 101 ordered ranks (0%, 1%, ..., 100%).

The Architecture

The APN consists of a backbone (like I3D or ResNet-50) followed by two "sibling heads":

  1. Classification Head (Cls): Predicts the action category.
  2. Localization Head (Loc): Predicts the current "progression stage" of the action.

Ordinal Regression

Crucially, the authors treat localization as an ordinal regression problem rather than a standard classification task. They found that a "Threshold Model" (specifically the CORAL method) performed best. Unlike standard classification, which treats "Stage 1" and "Stage 99" as equally different from "Stage 2," ordinal regression respects the numerical order, helping the model learn the linear evolution of an action.

Inference via Profile-Matching

During testing, the model generates a sequence of progression scores for the video. A simple profile-matching algorithm then looks for sub-sequences that follow a near-linear path from 0 to 100. If a segment matches this "ideal" progression profile, it is flagged as a completed action.

Datasets

The researchers evaluated their model on two primary benchmarks:

  • THUMOS14: A standard, challenging benchmark for TAD containing 20 categories of human actions.
  • DFMAD70: A new dataset introduced by the authors focusing on children's behavior. It features much longer actions (average 31 seconds vs. THUMOS14's few seconds) and strict annotations requiring complete action instances.

Key Insights and Ideas

  • Granularity Matters: Moving from 3 stages to 101 stages allows for much finer boundary detection. The "aha moment" is that actions have a predictable "rhythm" that can be mapped to a linear scale.
  • Order is Information: By using ordinal regression, the model learns that stages are related. This prevents the "inconsistency problem" where a model might jump erratically between progression percentages.
  • Training on "Clean" Data: The APN is trained exclusively on action frames (trimmed videos). This makes it surprisingly robust to diverse background environments in untrimmed test videos because the model only learns what the action looks like, not what the background looks like.
  • Efficiency: Because the localization logic is shifted into a simple profile-matching algorithm post-inference, the neural network itself remains lightweight and fast.

Key Results

  • THUMOS14: Achieved a mean Average Precision (mAP) of 58.3% at an IoU threshold of 0.5. This is competitive with state-of-the-art methods while being significantly simpler to train.
  • DFMAD70: On this long-duration dataset, the APN dominated with 98.9% mAP, vastly outperforming traditional methods (which hovered around 78%).
  • Speed: The 2D version of the model (APN-2D) reached an inference speed of 6277 FPS on a single 2080Ti GPU, making it viable for real-time applications.
  • Long-Action Specialist: The model showed a massive precision boost (up to 85% vs 20% for competitors) on actions with clear evolution patterns like "High Jump" or "Diving."

Limitations and Future Work

The authors note that the APN is specifically designed for "progressive actions"—those with a clear beginning, middle, and end. It struggles with:

  • Indeterminate actions: Actions like "Billiards" where the visual state doesn't change linearly.
  • Repetitive actions: Actions like "Walking" where the progression is cyclical rather than a single 0-100% arc.
  • Ephemeral actions: Very short bursts like a "Cricket Shot" (15.4% precision) where there isn't enough temporal "room" to map 101 stages effectively.

Future work may involve combining this progression-guided approach with other techniques to better handle these non-linear or repetitive action types.

Summary updated

Abstract

We present a novel framework, Action Progression Network (APN), for temporal action detection (TAD) in videos. The framework locates actions in videos by detecting the action evolution process. To encode the action evolution, we quantify a complete action process into 101 ordered stages (0\%, 1\%, ..., 100\%), referred to as action progressions. We then train a neural network to recognize the action progressions. The framework detects action boundaries by detecting complete action processes in the videos, e.g., a video segment with detected action progressions closely follow the sequence 0\%, 1\%, ..., 100\%. The framework offers three major advantages: (1) Our neural networks are trained end-to-end, contrasting conventional methods that optimize modules separately; (2) The APN is trained using action frames exclusively, enabling models to be trained on action classification datasets and robust to videos with temporal background styles differing from those in training; (3) Our framework effectively avoids detecting incomplete actions and excels in detecting long-lasting actions due to the fine-grained and explicit encoding of the temporal structure of actions. Leveraging these advantages, the APN achieves competitive performance and significantly surpasses its counterparts in detecting long-lasting actions. With an IoU threshold of 0.5, the APN achieves a mean Average Precision (mAP) of 58.3\% on the THUMOS14 dataset and 98.9\% mAP on the DFMAD70 dataset.
BibTeX
@misc{lu2023progression,
  author = {Chongkai Lu and Man‐Wai Mak and Ruimin Li and Zheru Chi and Hong Fu},
  title = {{Progression-Guided Temporal Action Detection in Videos}},
  year = {2023},
  eprint = {2308.09268},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.09268},
}
Figure