Explicit Motion Disentangling for Efficient Optical Flow Estimation

Changxing Deng, Ao Luo, Haibin Huang, Shaodan Ma, Jiangyu Liu, Shuaicheng LiuICCV 2023 · DOI

Abstract

EMD-Flow introduces a novel framework for optical flow estimation that explicitly disentangles global motion matching from local refinement. By addressing the computational inefficiencies of the dominant RAFT paradigm, EMD-Flow achieves state-of-the-art accuracy—ranking #1 on the KITTI-2015 benchmark—while being up to 85% faster than high-performance models like FlowFormer.

Introduction

Optical flow estimation, the task of predicting 2D pixel motion between video frames, is a cornerstone of computer vision. Currently, the field is led by the RAFT (Recurrent All-Pairs Field Transforms) architecture, which utilizes a heavy, iterative 4D cost-volume decoder. However, the authors identify a significant inefficiency: RAFT applies the same expensive machinery to both large global displacements and minor local refinements.

Analysis of motion and time distribution in RAFT vs. EMD-Flow.

Analysis of RAFT's iterations reveals that while early steps handle large motions, subsequent iterations primarily process small-scale refinements, leading to significant computational waste. As shown in the time distribution analysis, the iterative process consumes the vast majority of inference time, which EMD-Flow aims to optimize.

Methodology

The proposed Explicit Motion Disentangling (EMD) framework splits optical flow estimation into two specialized stages:

1. Global Matching

To resolve ambiguities from repetitive textures or large motions, the Multi-scale Motion Aggregation (MMA) module aggregates matches across scales (1/8, 1/16, and 1/32). This is followed by Confidence-induced Flow Propagation (CFP), which generates a confidence map to guide flow propagation in unreliable regions, such as occlusions, using a transformer-like mechanism.

2. Local Refinement

With a robust initial flow established, the model uses a lightweight recurrent decoder focused solely on sub-pixel accuracy. This "thinner" refinement process avoids the overhead of massive 4D cost-volume searches at every step. The authors introduce three variants: EMD-S (Small), EMD-M (Medium), and EMD-L (Large).

Experimental Results

EMD-Flow was evaluated on standard benchmarks including FlyingChairs, MPI Sintel, and KITTI-2015. EMD-L achieved an F1-all score of 4.51 on KITTI-2015, securing the top rank on the official leaderboard at the time of submission.

Performance comparison (F1-all vs. Inference Time) of EMD-Flow variants against existing methods.

As illustrated in the performance comparison, EMD-Flow variants significantly outperform existing methods in the accuracy-speed trade-off. EMD-L is 85% faster than FlowFormer (0.120s vs 0.930s) while using fewer parameters. Even the lightweight EMD-S model matches RAFT's accuracy on Sintel while running in just one-third of the time. This efficiency stems from the model's ability to converge in fewer iterations due to its superior global matching initialization.

Conclusion

EMD-Flow demonstrates that explicit motion disentangling is a powerful strategy for efficient optical flow. By separating global matching from local refinement, the framework achieves superior accuracy with a fraction of the computational cost. Future work will explore enhancing the robustness of the initial global matching stage and extending the framework to leverage temporal consistency in video sequences for further stability.

Summary updated

BibTeX
@inproceedings{deng2023explicit,
  author = {Changxing Deng and Ao Luo and Haibin Huang and Shaodan Ma and Jiangyu Liu and Shuaicheng Liu},
  title = {{Explicit Motion Disentangling for Efficient Optical Flow Estimation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00873},
}
Figure