Separable Flow: Learning Motion Cost Volumes for Optical Flow Estimation

Feihu Zhang, Oliver J. Woodford, Victor Prisacariu, Philip H. S. TorrICCV 2021 · OpenReview

Introduction

Optical flow estimation—the task of determining per-pixel 2D motion between consecutive video frames—is a fundamental challenge in computer vision. While modern deep learning architectures have made significant strides, they often rely on 4D cost volumes that represent matching costs across a full 2D search space. This approach introduces two primary difficulties: high computational complexity that scales quadratically with motion range, and local matching ambiguities in regions with occlusions or minimal texture where local feature correlations are insufficient.

The Separable Flow Module

To address these challenges, we introduce Separable Flow, a novel module designed to decompose the massive 4D cost volume into two efficient 3D volumes representing horizontal ($u$) and vertical ($v$) motion. This decomposition is achieved through three integrated stages:

  1. Self-adaptive Cost Separation: An attention-based mechanism compresses the 4D volume into two 3D feature tensors, reducing complexity from quadratic to linear relative to the motion range.
  2. Non-local Cost Aggregation: By operating on 1D components, the module can leverage Semi-Global Aggregation (SGA) layers. These layers facilitate information flow across the entire image, allowing the model to resolve local ambiguities by incorporating global scene structure and neighboring pixel motion.
  3. Motion Regression: Unlike previous frameworks that initialize iterative refinement at zero motion, Separable Flow employs a regression layer to predict a high-quality initial flow from the aggregated volumes, providing a superior starting point for subsequent refinement.
Figure 3: Comparison of flow estimation and matching score surfaces between RAFT and the proposed Separable Flow method. The matching score surface for Separable Flow (bottom right) shows a much clearer peak compared to the noisy surface of RAFT (bottom left).

Experimental Results

The model was evaluated on the Sintel (Clean and Final) and KITTI-2015 benchmarks after training on synthetic datasets like FlyingChairs and FlyingThings3D. Separable Flow achieved state-of-the-art performance, ranking 1st on both benchmarks at the time of publication. Notably, it reduced the End-Point Error (EPE) by 7% on Sintel and the outlier rate by 9% on KITTI compared to the previous state-of-the-art, RAFT.

A key strength of the proposed method is its superior generalization. When trained solely on synthetic data and tested on the real-world KITTI dataset, Separable Flow achieved a 15.9% error rate, significantly outperforming RAFT's 17.4%. This suggests that learning to aggregate costs globally is more robust across domains than relying solely on local feature matching.

Key Insights and Conclusion

The success of Separable Flow stems from the observation that 2D motion regression is naturally separable. By refining the cost volume itself rather than just the output flow, the model effectively eliminates false matching peaks caused by noise or occlusions. Despite the addition of sophisticated aggregation layers, the model remains efficient, maintaining a parameter count (~6.0M) and inference speed (~0.25s) comparable to RAFT.

While the current approach significantly improves flow estimation, occlusions remain a challenge where objects visible in only one frame are often predicted as stationary. Future research may explore multi-view or video-based inputs to better handle these temporal discontinuities.

Summary updated

BibTeX
@inproceedings{zhang2021separable,
  author = {Feihu Zhang and Oliver J. Woodford and Victor Prisacariu and Philip H. S. Torr},
  title = {{Separable Flow: Learning Motion Cost Volumes for Optical Flow Estimation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=8wqPMgFcm4m},
}
Figure