MotionSqueeze: Neural Motion Feature Learning for Video Understanding

Heeseung Kwon, Manjin Kim, Suha Kwak, Minsu Cho

Abstract

Video understanding requires capturing temporal dynamics, traditionally achieved through expensive external optical flow calculations or computationally heavy 3D CNNs. We present MotionSqueeze (MS), a lightweight, plug-and-play neural module that enables video recognition models to learn motion features internally. By replacing external flow with a differentiable "correlation-to-displacement" pipeline, MSNet achieves state-of-the-art accuracy on motion-heavy datasets with negligible computational overhead.

Introduction

In computer vision, images describe static objects, but videos describe actions. To understand these actions, models must capture motion. Historically, researchers have used two-stream networks (relying on slow, pre-calculated optical flow) or 3D CNNs (which are computationally massive and often struggle with relative movement). MotionSqueeze identifies this "computational bottleneck" and proposes a method to gain the accuracy of motion-aware features without the heavy lifting of external flow or the brute force of massive convolutions.

The MotionSqueeze Module

The core of our approach is the MotionSqueeze (MS) module, a trainable block that can be inserted into existing CNN architectures. The module operates in three distinct stages to "squeeze" out motion information from adjacent frames.

Detailed architecture of the MotionSqueeze (MS) module, showing the correlation computation, displacement estimation, and feature transformation stages.
  1. Correlation Computation: The module calculates a "correlation score" for pixels in a local neighborhood between adjacent feature maps, identifying where features have moved.
  2. Displacement Estimation: To maintain end-to-end differentiability, a kernel-soft-argmax operation is used. This calculates a weighted average of possible movements rather than a discrete match, allowing the network to learn motion through standard backpropagation.
  3. Feature Transformation: Raw geometric displacement maps are processed through depth-wise separable convolutions to produce high-level motion features suitable for classification.

Experimental Results

MSNet was evaluated on standard benchmarks including Something-Something (V1 & V2), Kinetics-400, and HMDB51. The results demonstrate that the MS module provides a significant performance boost with minimal cost.

Comparison of Top-1 Accuracy vs. FLOPs and Parameter count. MSNet (Ours) achieves superior accuracy with significantly lower computational cost compared to other state-of-the-art methods.

Key findings include:

  • Efficiency: Adding the MS module increases FLOPs by only 2.5% and parameters by 1.2%.
  • Speed: MSNet is approximately 160x faster than traditional two-stream networks that rely on iterative optical flow.
  • Performance: On the motion-centric Something-Something V1 dataset, MSNet achieved 55.1% top-1 accuracy, outperforming existing methods and providing a substantial jump over the Temporal Shift Module (TSM) baseline.

Conclusion

MotionSqueeze demonstrates that explicit motion modeling can be both effective and efficient. By internalizing motion learning, the network "discovers" how to track objects across frames to minimize classification error. While highly effective, future work will focus on improving robustness in scenarios with severe occlusions and exploring adaptive neighborhoods to capture a wider range of motion scales.

Summary updated

BibTeX
@inproceedings{kwon2020motionsqueeze,
  author = {Heeseung Kwon and Manjin Kim and Suha Kwak and Minsu Cho},
  title = {{MotionSqueeze: Neural Motion Feature Learning for Video Understanding}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58517-4\_21},
}
Figure