MPI-Flow: Learning Realistic Optical Flow with Multiplane Images

Yingping Liang, Jiaming Liu, Debing Zhang, Ying Fu

Abstract

Training deep learning models for optical flow is traditionally limited by the scarcity of real-world ground truth data. While synthetic datasets provide perfect labels, they often fail to capture the complexity of real-world environments. MPI-Flow addresses this challenge by using Multiplane Images (MPI) to generate high-fidelity training data from single static photos. By decoupling camera motion from independent object motion and utilizing depth-aware inpainting, MPI-Flow creates realistic video pairs with accurate flow labels. The method significantly outperforms previous synthetic and real-image-based approaches on major benchmarks, including KITTI and Sintel.

Introduction

Optical flow—the task of tracking pixel-level motion between consecutive video frames—is fundamental to computer vision applications such as autonomous driving and video editing. However, obtaining "ground truth" labels for real-world sequences is nearly impossible. Historically, researchers have relied on synthetic datasets like FlyingChairs, which, despite providing perfect labels, suffer from a significant "domain gap" when applied to real-world scenes.

Existing attempts to bridge this gap, such as Depthstillation or RealFlow, warp real images to create training pairs. However, these methods frequently introduce "ghosting" artifacts and blurriness, particularly at object boundaries. Furthermore, they often struggle to model independent object motion (IOM), assuming a static scene where only the camera moves. MPI-Flow proposes a 3D-aware pipeline to overcome these limitations.

Methodology

MPI-Flow moves beyond 2D pixel warping by adopting a Multiplane Image (MPI) representation. This approach constructs a stack of semi-transparent layers at varying depths from a single input image.

MPI Construction and Volume Rendering

From a single "seed" image, the system estimates depth and builds the MPI layers. Instead of flat 2D warping, the system manipulates these 3D planes based on virtual camera trajectories. Volume rendering is then used to project these planes back into a 2D frame. This process naturally handles occlusion and depth discontinuities, preventing the "holes" typically seen in 2D warping methods.

Independent Object Motion (IOM)

A key innovation of MPI-Flow is the explicit modeling of moving objects. Using instance segmentation, the system identifies and "cuts out" specific objects, such as vehicles or pedestrians.

Scene segmentation for Independent Object Motion (IOM), where objects are decoupled from the background for separate motion modeling.

By applying distinct motion vectors to these objects relative to the background, the pipeline generates training data that reflects the dynamic nature of real-world traffic and street scenes.

Depth-Aware Inpainting

When an object moves, it reveals previously occluded background areas. MPI-Flow employs a specialized depth-aware inpainting module to fill these regions realistically. This ensures that the synthesized frames maintain the visual consistency of real photographs rather than appearing as digital collages.

Experiments and Results

The researchers evaluated MPI-Flow using images from COCO and DAVIS to generate large-scale training sets, testing the resulting models on the Sintel and KITTI benchmarks.

Quantitative Performance

MPI-Flow sets a new state-of-the-art for unsupervised optical flow on the KITTI 2015 benchmark, achieving an Outlier Ratio (Fl) of 6.63%, showing strong performance against established methods like UFlow (9.05%). When used to fine-tune the popular RAFT model, MPI-Flow reached an End-Point Error (EPE) of 1.88 on KITTI 15, outperforming models trained on other real-image generators.

Qualitative Comparison

Visual analysis demonstrates that MPI-Flow produces significantly cleaner images with sharper boundaries compared to Depthstillation and RealFlow.

Qualitative comparison of MPI-Flow against existing methods, showing superior detail and reduced artifacts in synthesized frames and flow maps.

Discussion and Conclusion

While MPI-Flow represents a significant advancement, it currently treats objects as rigid bodies, failing to account for non-rigid motions like human gait. Additionally, the quality of the generated flow is dependent on the accuracy of the underlying depth estimation and segmentation models. Future work will focus on incorporating non-rigid motion and reducing the computational overhead associated with MPI volume rendering.

Despite these limitations, MPI-Flow demonstrates that a single high-quality photograph contains sufficient geometric information to synthesize complex, realistic motion sequences for training robust optical flow models.

The code is available on GitHub at Sharpiless/MPI-Flow.

Summary updated

Abstract

The accuracy of learning-based optical flow estimation models heavily relies on the realism of the training datasets. Current approaches for generating such datasets either employ synthetic data or generate images with limited realism. However, the domain gap of these data with real-world scenes constrains the generalization of the trained model to real-world applications. To address this issue, we investigate generating realistic optical flow datasets from real-world images. Firstly, to generate highly realistic new images, we construct a layered depth representation, known as multiplane images (MPI), from single-view images. This allows us to generate novel view images that are highly realistic. To generate optical flow maps that correspond accurately to the new image, we calculate the optical flows of each plane using the camera matrix and plane depths. We then project these layered optical flows into the output optical flow map with volume rendering. Secondly, to ensure the realism of motion, we present an independent object motion module that can separate the camera and dynamic object motion in MPI. This module addresses the deficiency in MPI-based single-view methods, where optical flow is generated only by camera motion and does not account for any object movement. We additionally devise a depth-aware inpainting module to merge new images with dynamic objects and address unnatural motion occlusions. We show the superior performance of our method through extensive experiments on real-world datasets. Moreover, our approach achieves state-of-the-art performance in both unsupervised and supervised training of learning-based models. The code will be made publicly available at: \url{https://github.com/Sharpiless/MPI-Flow}.
BibTeX
@inproceedings{liang2023mpi,
  author = {Yingping Liang and Jiaming Liu and Debing Zhang and Ying Fu},
  title = {{MPI-Flow: Learning Realistic Optical Flow with Multiplane Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure