Abstract
Action detection in video requires identifying both the category and the spatio-temporal extent of actions. Traditional methods often rely on offline processing or separate stream training, limiting real-time applicability. We present Two-Stream AMTnet, an end-to-end trainable framework that detects actions in "micro-tubes"—pairs of bounding boxes linked across time. By integrating appearance and motion streams through early fusion, the model achieves state-of-the-art accuracy on J-HMDB-21 and UCF-101-24 benchmarks while maintaining real-time inference speeds of 21-33 fps.
Introduction
Real-time action detection is critical for autonomous systems and human-robot interaction. However, existing detectors face several challenges. First, many are "frame-centric," treating video as independent images and failing to capture temporal dynamics. Second, "late fusion" approaches train appearance (RGB) and motion (optical flow) networks separately, missing the opportunity to learn cross-modal interactions. Finally, many models are "offline," requiring the full video sequence before generating action tubes. Two-Stream AMTnet addresses these by proposing an online, end-to-end framework that is robust to sparse annotations and operates in real-time.
Methods
The core of the architecture is the Action Micro-Tube (AMTnet). Instead of predicting a single bounding box per frame, the network predicts a micro-tube: a pair of bounding boxes across two frames ($f_t$ and $f_{t+\Delta}$).
Architecture and Fusion
The model utilizes two parallel VGG-16-based streams. The appearance stream processes RGB pairs, while the motion stream processes stacks of optical flow maps. A critical innovation is train-time fusion, where features from both streams are merged (via element-wise summation or concatenation) within the network. This allows the model to learn pixel-wise correspondences between appearance and motion. To ensure efficiency, the framework adopts a Single Shot MultiBox Detector (SSD) backbone.
Online Tube Generation
To handle streaming video, an incremental algorithm links micro-tubes as they are generated. This linking process uses Intersection over Union (IoU) and class confidence scores to build continuous action tubes over time.
Experimental Results
The model was evaluated on the J-HMDB-21 and UCF-101-24 datasets.
- Accuracy: On J-HMDB-21, the "sum fusion" variant achieved a video-mAP improvement of 7.6% at high overlap thresholds (0.75 IoU) over previous state-of-the-art methods. It also set new performance benchmarks on the untrimmed UCF-101-24 dataset.
- Temporal Robustness: The model maintains high accuracy even with large frame gaps ($\Delta = 16$), demonstrating robustness to sparse data.
- Efficiency: The two-stream model operates at 21 fps on a single NVIDIA 1080Ti. Using only the appearance stream increases throughput to 33 fps, enabling real-time deployment.
Conclusion
Two-Stream AMTnet demonstrates that early fusion of appearance and motion, combined with a micro-tube representation, significantly enhances both the accuracy and speed of action detection. While optical flow computation remains a bottleneck, the framework's ability to operate online and handle sparse annotations makes it a viable solution for real-world applications. Future work will focus on integrating more efficient motion estimation and leveraging scene context to improve detection in complex environments.
Summary updated