Abstract
Interactive autonomous applications require robustness of the perception engine to artifacts in unconstrained videos. In this paper, we examine the effect of camera motion on the task of action detection. We develop a novel ranking method to rank videos based on the degree of global camera motion. For the high ranking camera videos we show that the accuracy of action detection is decreased. We propose an action detection pipeline that is robust to the camera motion effect and verify it empirically. Specifically, we do actor feature alignment across frames and couple global scene features with local actor-specific features. We do feature alignment using a novel formulation of the Spatio-temporal Sampling Network (STSN) but with multi-scale offset prediction and refinement using a pyramid structure. We also propose a novel input dependent weighted averaging strategy for fusing local and global features. We show the applicability of our network on our dataset of moving camera videos with high camera motion (MOVE dataset) with a 4.1% increase in frame mAP and 17% increase in video mAP.
Introduction
In autonomous systems like self-driving cars and mobile robots, perception engines must do more than identify objects; they must interpret actions in real-time. However, most contemporary action detection models are trained on datasets characterized by static or smooth camera movements. When these models are deployed on moving platforms, they encounter significant ego-motion—jitter, vibration, and motion blur—that leads to "feature misalignment." In such scenarios, the spatial representation of an actor in one frame does not align with subsequent frames, causing standard deep learning architectures to fail at linking features temporally. RADNet addresses this by introducing a framework robust to arbitrary camera motion.
Proposed Methodology
RADNet enhances the Action Tubelet (ACT) framework through two primary architectural innovations designed to counteract motion-induced degradation.
Multi-Scale Feature Alignment
To resolve misalignment, RADNet utilizes deformable convolutions to realign features from neighboring frames to a reference frame. Unlike traditional Spatio-temporal Sampling Networks (STSN) that predict offsets at a single scale, RADNet employs a pyramid refinement structure. It predicts offsets across multiple scales (from 38x38 down to 1x1). Coarse offsets from the top of the pyramid provide a global motion estimate, which is then upsampled and refined at finer scales to capture local actor movements. This allows the model to compensate for both large-scale camera shifts and subtle subject motion.
Dynamic Global-Local Fusion
While action detection typically prioritizes local actor features, extreme camera motion can degrade these features significantly. RADNet introduces a Weighted Averaging strategy to fuse local actor features with global scene context. Using a sub-network called EmbedNet, the model dynamically assigns weights to these features based on the input quality. If local features are deemed too noisy due to blur, the model leans more heavily on the stable global context to maintain classification accuracy.
The MOVE Dataset and Ranking Metric
To evaluate robustness, the authors introduced the MOVE dataset, consisting of 43 high-motion sequences from YouTube across 15 action classes. To quantify the difficulty of these videos, a novel ranking method was developed.
As shown in the figure above, the system calculates the average magnitude of dense optical flow (Brox Flow) while masking out annotated actors. This isolates global camera motion from actor motion, allowing for a quantitative ranking of video "jitter." This metric confirms that the MOVE dataset presents a significantly higher challenge than standard benchmarks like UCF101.
Experimental Results
RADNet demonstrated substantial improvements over the baseline ACT ResNet-50 on high-motion data:
- MOVE Dataset: Achieved a 4.1% increase in frame mAP and a 17% increase in video mAP (at a 0.3 threshold).
- Ablation Studies: The pyramid refinement module was identified as the most critical component, contributing a 9% boost in video mAP over standard STSN alignment.
- Generalization: On the JHMDB-21 benchmark, RADNet achieved 75.1% mAP, outperforming state-of-the-art models like ROAD and TACNet, indicating that its motion-robustness does not compromise performance on stable video.
Conclusion
RADNet provides a robust solution for action detection in moving autonomous systems. By combining multi-scale feature alignment with intelligent global-local feature fusion, it effectively mitigates the negative impacts of ego-motion. Future work will focus on expanding the MOVE dataset for training purposes and exploring two-stream architectures that explicitly incorporate optical flow as an input modality.
Summary updated