Prior-Enhanced Temporal Action Localization Using Subject-Aware Spatial Attention

Yifan Liu, Youbao Tang, Ning Zhang, Ruei-Sung Lin, Haoqian Wang

Abstract

Temporal action localization (TAL) aims to detect the boundary and identify the class of each action instance in a long untrimmed video. Current approaches treat video frames homogeneously, and tend to give background and key objects excessive attention. This limits their sensitivity to localize action boundaries. To this end, we propose a prior-enhanced temporal action localization method (PETAL), which only takes in RGB input and incorporates action subjects as priors. This proposal leverages action subjects' information with a plug-and-play subject-aware spatial attention module (SA-SAM) to generate an aggregated and subject-prioritized representation. Experimental results on THUMOS-14 and ActivityNet-1.3 datasets demonstrate that the proposed PETAL achieves competitive performance using only RGB features, e.g., boosting mAP by 2.41% over the state-of-the-art approach that uses RGB features on the THUMOS-14 dataset.

Introduction: The Challenge of Action Boundaries

Temporal Action Localization (TAL) requires pinpointing the exact start and end times of actions within untrimmed video streams. A significant hurdle in current TAL research is the "homogeneous frame processing" paradigm, where models assign equal importance to all spatial regions in a frame.

This often leads to poor boundary sensitivity. For instance, a model might incorrectly classify a segment as "sliding" simply because a slide is visible in the background, even if no person is actively using it. This spatial ambiguity results in noisy detections where predicted action boundaries bleed into inactive segments.

Illustration of the boundary detection challenge where background objects cause false positives.

Proposed Method: PETAL

To address these limitations, we introduce PETAL (Prior-Enhanced Temporal Action Localization). PETAL shifts the focus from global frame features to specific action subjects, typically humans, using them as spatial priors to guide the localization process.

The PETAL architecture featuring SA-DRM, SA-SAM, and H-TAM modules.

The architecture is built around three core components:

  1. Subject-aware Detection Ranking Module (SA-DRM): PETAL utilizes a pre-trained detector (e.g., YOLOX) to identify potential subjects. These detections are ranked by confidence and size, and the top $K$ subjects are selected as the primary focus for the current frame.
  2. Subject-aware Spatial Attention Module (SA-SAM): This module processes "Individual Tokens" extracted from the $K$ subjects. Through multi-head self-attention (MHSA), it aggregates these features into a single "Group Token." This mechanism ensures the model prioritizes the actors' movements over static background elements.
  3. Hierarchical Temporal Attention Module (H-TAM): The sequence of group tokens is processed through a hierarchical transformer to capture temporal dependencies at multiple scales. This allows for precise boundary regression and action classification across varying action durations.

Experimental Results

We evaluated PETAL on the THUMOS-14 and ActivityNet-1.3 benchmarks using VideoMAE features extracted solely from RGB frames.

Performance on THUMOS-14

On the THUMOS-14 dataset, PETAL achieved a mean Average Precision (mAP) of 67.05%. This represents a 2.41% improvement over the previous state-of-the-art RGB-based model (ActionFormer). Notably, PETAL's RGB-only performance even surpasses some existing methods that utilize both RGB and computationally expensive optical flow features.

Performance on ActivityNet-1.3

On the larger ActivityNet-1.3 dataset, PETAL reached 35.34% mAP, demonstrating its scalability and effectiveness across a wider variety of action classes (200 vs. 20 in THUMOS-14).

Ablation Studies

Our analysis showed that focusing on the top 6 subjects ($K=6$) yields optimal performance. Furthermore, increasing the depth of the SA-SAM module to 8 layers consistently improved the model's ability to resolve complex scenes with multiple actors.

Conclusion

PETAL demonstrates that incorporating subject-aware spatial priors significantly enhances temporal action localization. By prioritizing the "who" in a scene, the model achieves superior boundary precision without the need for explicit motion data like optical flow. While currently dependent on the quality of the initial subject detector, PETAL's plug-and-play nature offers a promising foundation for future multi-modal TAL frameworks.

Summary updated

Abstract

Temporal action localization (TAL) aims to detect the boundary and identify the class of each action instance in a long untrimmed video. Current approaches treat video frames homogeneously, and tend to give background and key objects excessive attention. This limits their sensitivity to localize action boundaries. To this end, we propose a prior-enhanced temporal action localization method (PETAL), which only takes in RGB input and incorporates action subjects as priors. This proposal leverages action subjects’ information with a plug-and-play subject-aware spatial attention module (SA-SAM) to generate an aggregated and subject-prioritized representation. Experimental results on THUMOS-14 and ActivityNet-1.3 datasets demonstrate that the proposed PETAL achieves competitive performance using only RGB features, e.g., boosting mAP by 2.41% or 0.25% over the state-of-the-art approach that uses RGB features or with additional optical flow features on the THUMOS-14 dataset.
BibTeX
@misc{liu2022prior,
  author = {Yifan Liu and Youbao Tang and Ning Zhang and Ruei-Sung Lin and Haoqian Wang},
  title = {{Prior-Enhanced Temporal Action Localization Using Subject-Aware Spatial Attention}},
  year = {2022},
  eprint = {2211.05299},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.05299},
}
Figure