A*: Atrous Spatial Temporal Action Recognition for Real Time Applications

Myeongjun Kim, Federica Spinola, Philipp Benz, Tae-hoon KimWACV 2024

TL;DR

This paper introduces A, a novel single-stage action recognition system designed for real-time applications. By integrating atrous convolutions for multi-scale spatio-temporal feature extraction and a new image-video feature fusion module, A achieves state-of-the-art performance on challenging benchmarks while maintaining high inference speeds, making it suitable for real-world scenarios like autonomous driving.

What Problem Does This Paper Address?

Recognizing human actions in videos is crucial for applications ranging from autonomous driving to surveillance. While deep learning has made significant strides, achieving accurate action recognition in real-time remains a challenge. Existing methods often struggle to balance capturing both fine-grained local details and broader global context, as well as temporal dependencies across frames. Furthermore, many high-performing models are computationally expensive, making them unsuitable for applications requiring immediate predictions without access to future frames. This paper aims to bridge this gap by proposing a system that is both accurate and fast, capable of real-time action recognition.

Methods

The proposed A* system builds upon the YOLO framework, adapting it for video analysis. It introduces four key modules:

  • Low-Level Feature Aggregation (LLFA): This module combines low-level and high-level features extracted from different stages of a 3D backbone. This is crucial for preserving spatial context that can be lost in deeper network layers, thereby capturing global scene information.
  • Atrous Spatio-Temporal Pyramid Pooling (ASTPP): Extending the concept of Atrous Spatial Pyramid Pooling (ASPP) to 3D, this module uses dilated convolutions with varying dilation rates along both spatial and temporal axes. This allows the network to capture multi-range temporal and long-range spatial information effectively, addressing the limitations of standard convolutions that operate at a single scale.
  • Image-Video Feature Fusion (IVFF): This module fuses keyframe-specific features (extracted via slicing from the LLFA output) with aggregated video features. This fusion helps to focus on the target frame while still leveraging the temporal context, mitigating noise from scene transitions or irrelevant frames. A self-attention mechanism is employed here to emphasize and combine global and local information.
  • Proxy Anchor Loss (PAL): To better distinguish between temporally ambiguous actions (e.g., sitting vs. standing), the paper incorporates Proxy Anchor Loss. This metric learning-based loss function helps the model learn more discriminative features for similar-looking actions.

The overall architecture is designed as a single-stage detector, enabling end-to-end training and faster inference.

Datasets

The paper evaluates A* on three widely-used action recognition benchmarks:

  • J-HMDB-21: A subset of HMDB-51, specifically curated for action recognition with 21 classes.
  • UCF101-24: A dataset designed for spatio-temporal action detection, featuring 24 action classes related to sports.
  • AVA (v2.1 and v2.2): A large-scale, multi-class dataset with 80 action classes, designed for realistic scenarios involving multiple people and actions.

Key Insights and Ideas

  • The importance of multi-scale spatio-temporal context: The ASTPP module's success highlights that capturing information at various temporal and spatial scales is vital for robust action recognition, going beyond what standard convolutions can achieve.
  • Leveraging low-level features for global context: The LLFA module demonstrates that incorporating low-level features, often discarded in deep networks, is essential for understanding the broader scene context, which can significantly aid action interpretation.
  • Keyframe focus with temporal awareness: The IVFF module's approach of extracting keyframe features and fusing them with temporal context proves effective in focusing predictions while still benefiting from the video sequence.
  • Addressing temporal ambiguity with metric learning: The integration of Proxy Anchor Loss effectively tackles the challenge of distinguishing between actions that appear similar when temporal order is reversed or ambiguous.

Key Results

A achieves state-of-the-art (SOTA) performance on the J-HMDB-21 and UCF101-24 datasets, outperforming previous methods by significant margins (e.g., +3.7 frame-mAP over STMixer on J-HMDB and +4.4 over HIT on UCF101-24). It also demonstrates competitive performance on the AVA dataset. Crucially, A achieves real-time inference speeds of up to 33 FPS on an RTX 3090 GPU, making it practical for real-world applications.

Limitations and Future Work

The authors acknowledge that for the AVA dataset, which involves multi-label and multi-person scenarios, image resolution plays a more critical role than in single-person datasets. They speculate that using square images in A* might lead to a loss of fine-grained details compared to methods that maintain aspect ratios. Future work will focus on developing networks that can "remember" past experiences and "infer" future frames from a given frame to further enhance real-time action detection performance. The paper suggests that further hyperparameter tuning, particularly for dilation sizes in ASTPP, could yield even better results.

Summary updated

BibTeX
@inproceedings{kim2024atrous,
  author = {Myeongjun Kim and Federica Spinola and Philipp Benz and Tae-hoon Kim},
  title = {{A*: Atrous Spatial Temporal Action Recognition for Real Time Applications}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure