Abstract
Video understanding requires capturing temporal dynamics, traditionally achieved through expensive external optical flow calculations or computationally heavy 3D CNNs. We present MotionSqueeze (MS), a lightweight, plug-and-play neural module that enables video recognition models to learn motion features internally. By replacing external flow with a differentiable "correlation-to-displacement" pipeline, MSNet achieves state-of-the-art accuracy on motion-heavy datasets with negligible computational overhead.
Introduction
In computer vision, images describe static objects, but videos describe actions. To understand these actions, models must capture motion. Historically, researchers have used two-stream networks (relying on slow, pre-calculated optical flow) or 3D CNNs (which are computationally massive and often struggle with relative movement). MotionSqueeze identifies this "computational bottleneck" and proposes a method to gain the accuracy of motion-aware features without the heavy lifting of external flow or the brute force of massive convolutions.
The MotionSqueeze Module
The core of our approach is the MotionSqueeze (MS) module, a trainable block that can be inserted into existing CNN architectures. The module operates in three distinct stages to "squeeze" out motion information from adjacent frames.
- Correlation Computation: The module calculates a "correlation score" for pixels in a local neighborhood between adjacent feature maps, identifying where features have moved.
- Displacement Estimation: To maintain end-to-end differentiability, a kernel-soft-argmax operation is used. This calculates a weighted average of possible movements rather than a discrete match, allowing the network to learn motion through standard backpropagation.
- Feature Transformation: Raw geometric displacement maps are processed through depth-wise separable convolutions to produce high-level motion features suitable for classification.
Experimental Results
MSNet was evaluated on standard benchmarks including Something-Something (V1 & V2), Kinetics-400, and HMDB51. The results demonstrate that the MS module provides a significant performance boost with minimal cost.
Key findings include:
- Efficiency: Adding the MS module increases FLOPs by only 2.5% and parameters by 1.2%.
- Speed: MSNet is approximately 160x faster than traditional two-stream networks that rely on iterative optical flow.
- Performance: On the motion-centric Something-Something V1 dataset, MSNet achieved 55.1% top-1 accuracy, outperforming existing methods and providing a substantial jump over the Temporal Shift Module (TSM) baseline.
Conclusion
MotionSqueeze demonstrates that explicit motion modeling can be both effective and efficient. By internalizing motion learning, the network "discovers" how to track objects across frames to minimize classification error. While highly effective, future work will focus on improving robustness in scenarios with severe occlusions and exploring adaptive neighborhoods to capture a wider range of motion scales.
Summary updated