Two-Stream AMTnet for Action Detection

Suman Saha, Gurkirt Singh, Fabio CuzzolinarXiv 2020 · OpenReview

Abstract

Action detection in video requires identifying both the category and the spatio-temporal extent of actions. Traditional methods often rely on offline processing or separate stream training, limiting real-time applicability. We present Two-Stream AMTnet, an end-to-end trainable framework that detects actions in "micro-tubes"—pairs of bounding boxes linked across time. By integrating appearance and motion streams through early fusion, the model achieves state-of-the-art accuracy on J-HMDB-21 and UCF-101-24 benchmarks while maintaining real-time inference speeds of 21-33 fps.

Introduction

Real-time action detection is critical for autonomous systems and human-robot interaction. However, existing detectors face several challenges. First, many are "frame-centric," treating video as independent images and failing to capture temporal dynamics. Second, "late fusion" approaches train appearance (RGB) and motion (optical flow) networks separately, missing the opportunity to learn cross-modal interactions. Finally, many models are "offline," requiring the full video sequence before generating action tubes. Two-Stream AMTnet addresses these by proposing an online, end-to-end framework that is robust to sparse annotations and operates in real-time.

Methods

The core of the architecture is the Action Micro-Tube (AMTnet). Instead of predicting a single bounding box per frame, the network predicts a micro-tube: a pair of bounding boxes across two frames ($f_t$ and $f_{t+\Delta}$).

The Two-Stream AMTnet architecture featuring appearance and motion streams with early fusion.

Architecture and Fusion

The model utilizes two parallel VGG-16-based streams. The appearance stream processes RGB pairs, while the motion stream processes stacks of optical flow maps. A critical innovation is train-time fusion, where features from both streams are merged (via element-wise summation or concatenation) within the network. This allows the model to learn pixel-wise correspondences between appearance and motion. To ensure efficiency, the framework adopts a Single Shot MultiBox Detector (SSD) backbone.

Online Tube Generation

To handle streaming video, an incremental algorithm links micro-tubes as they are generated. This linking process uses Intersection over Union (IoU) and class confidence scores to build continuous action tubes over time.

Illustration of the micro-tube concept and temporal linking.

Experimental Results

The model was evaluated on the J-HMDB-21 and UCF-101-24 datasets.

  • Accuracy: On J-HMDB-21, the "sum fusion" variant achieved a video-mAP improvement of 7.6% at high overlap thresholds (0.75 IoU) over previous state-of-the-art methods. It also set new performance benchmarks on the untrimmed UCF-101-24 dataset.
  • Temporal Robustness: The model maintains high accuracy even with large frame gaps ($\Delta = 16$), demonstrating robustness to sparse data.
  • Efficiency: The two-stream model operates at 21 fps on a single NVIDIA 1080Ti. Using only the appearance stream increases throughput to 33 fps, enabling real-time deployment.

Conclusion

Two-Stream AMTnet demonstrates that early fusion of appearance and motion, combined with a micro-tube representation, significantly enhances both the accuracy and speed of action detection. While optical flow computation remains a bottleneck, the framework's ability to operate online and handle sparse annotations makes it a viable solution for real-world applications. Future work will focus on integrating more efficient motion estimation and leveraging scene context to improve detection in complex environments.

Summary updated

BibTeX
@misc{saha2020two,
  author = {Suman Saha and Gurkirt Singh and Fabio Cuzzolin},
  title = {{Two-Stream AMTnet for Action Detection}},
  year = {2020},
  url = {https://openreview.net/forum?id=4j6_ND3ef7r},
}
Figure