CFAD: Coarse-to-Fine Action Detector for Spatiotemporal Action Localization

Yuxi Li, Weiyao Lin, John See, Ning Xu, Shugong Xu, Yan Ke, Cong Yang

Abstract

Most current pipelines for spatio-temporal action localization connect frame-wise or clip-wise detection results to generate action proposals, where only local information is exploited and the efficiency is hindered by dense per-frame localization. In this paper, we propose Coarse-to-Fine Action Detector (CFAD), an original end-to-end trainable framework for efficient spatio-temporal action localization. The CFAD introduces a new paradigm that first estimates coarse spatio-temporal action tubes from video streams, and then refines the tubes' location based on key timestamps. This concept is implemented by two key components, the Coarse and Refine Modules in our framework. The parameterized modeling of long temporal information in the Coarse Module helps obtain accurate initial tube estimation, while the Refine Module selectively adjusts the tube location under the guidance of key timestamps. Against other methods, the proposed CFAD achieves competitive results on action detection benchmarks of UCF101-24, UCFSports and JHMDB-21 with inference speed that is 3.3x faster than the nearest competitors.

Spatiotemporal action localization requires identifying the "what," "where," and "when" of actions in video. Traditional methods typically follow a "detect-and-link" strategy: they perform spatial object detection on every frame and then link these boxes into a temporal "action tube." This approach is computationally expensive due to per-frame redundancy and often lacks global context, leading to jittery trajectories.

Comparison between (a) traditional detect-and-link and (b) the proposed coarse-to-fine paradigm.

CFAD proposes a paradigm shift. Instead of treating action tubes as a collection of discrete boxes, it treats them as continuous, parameterized trajectories. This allows for a more holistic and efficient "coarse-to-fine" approach that utilizes global temporal context.

Methodology

The CFAD framework is composed of two main stages: the Coarse Module for global estimation and the Refine Module for selective adjustment.

The CFAD architecture, featuring the Coarse Module for tube estimation and the Refine Module for key-timestamp-based adjustment.

1. The Coarse Module (Global Estimation)

The Coarse Module utilizes Parameterized Modeling to represent an action's trajectory (x, y, width, height) as a high-order polynomial function of time. By predicting polynomial coefficients ($\theta$) rather than thousands of individual coordinates, the model supervises the entire shape of the action tube simultaneously. This process leverages global temporal context from a 3D CNN and a "Diff Block" that processes temporal residuals to distinguish actions with similar static appearances but different motion signatures.

2. The Refine Module (Selective Adjustment)

To account for fine-grained details or sudden movements that a simple polynomial might miss, the Refine Module performs targeted adjustments:

  • Key Timestamp Selection: A 1D hourglass network identifies critical frames for localization.
  • Local Search & RPN: At these key timestamps, a class-specific Region Proposal Network (RPN) searches locally around the coarse estimate to find high-scoring proposals.
  • Interpolation: Final action tubes are generated by smoothing the refined boxes and coarse estimates using piece-wise cubic spline interpolation.

Experimental Results

CFAD was evaluated on the UCF101-24, JHMDB-21, and UCFSports benchmarks, demonstrating significant improvements in both speed and accuracy.

Performance and Efficiency

  • Speed: CFAD achieves an inference speed of 7.6ms per frame, making it 3.3x faster than the previous state-of-the-art (Singh et al., 25ms).
  • Accuracy: On UCF101-24, CFAD reached a video-mAP of 62.7% (at $\delta=0.5$), outperforming many complex two-stream models using only RGB input.
  • Ablation: The refinement process is highly effective; adding the Refine Module improved video-mAP on UCF101-24 by over 14% compared to using the Coarse Module alone.

Discussion and Insights

The core innovation of CFAD lies in the realization that smoothness is a prior for human motion. By baking this into the architecture via polynomials, the search space is drastically reduced. Efficiency is further gained through sparsity, focusing heavy-duty refinement only on key timestamps.

While highly effective, the method has limitations. The reliance on interpolation can sometimes lead to lower frame-level mAP compared to dense detection, and extremely erratic motion may challenge the polynomial modeling. Future work may involve better capturing motion signatures directly within the 3D CNN to further reduce dependency on optical flow.

Summary updated

Abstract

Most current pipelines for spatio-temporal action localization connect frame-wise or clip-wise detection results to generate action proposals, where only local information is exploited and the efficiency is hindered by dense per-frame localization. In this paper, we propose Coarse-to-Fine Action Detector (CFAD), an original end-to-end trainable framework for efficient spatio-temporal action localization. The CFAD introduces a new paradigm that first estimates coarse spatio-temporal action tubes from video streams, and then refines the tubes’ location based on key timestamps. This concept is implemented by two key components, the Coarse and Refine Modules in our framework. The parameterized modeling of long temporal information in the Coarse Module helps obtain accurate initial tube estimation, while the Refine Module selectively adjusts the tube location under the guidance of key timestamps. Against other methods, the proposed CFAD achieves competitive results on action detection benchmarks of UCF101-24, UCFSports and JHMDB-21 with inference speed that is 3.3 $$\times $$ faster than the nearest competitor.
BibTeX
@inproceedings{li2020cfad,
  author = {Yuxi Li and Weiyao Lin and John See and Ning Xu and Shugong Xu and Yan Ke and Cong Yang},
  title = {{CFAD: Coarse-to-Fine Action Detector for Spatiotemporal Action Localization}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58517-4\_30},
}
Figure