Abstract
Training deep learning models for optical flow is traditionally limited by the scarcity of real-world ground truth data. While synthetic datasets provide perfect labels, they often fail to capture the complexity of real-world environments. MPI-Flow addresses this challenge by using Multiplane Images (MPI) to generate high-fidelity training data from single static photos. By decoupling camera motion from independent object motion and utilizing depth-aware inpainting, MPI-Flow creates realistic video pairs with accurate flow labels. The method significantly outperforms previous synthetic and real-image-based approaches on major benchmarks, including KITTI and Sintel.
Introduction
Optical flow—the task of tracking pixel-level motion between consecutive video frames—is fundamental to computer vision applications such as autonomous driving and video editing. However, obtaining "ground truth" labels for real-world sequences is nearly impossible. Historically, researchers have relied on synthetic datasets like FlyingChairs, which, despite providing perfect labels, suffer from a significant "domain gap" when applied to real-world scenes.
Existing attempts to bridge this gap, such as Depthstillation or RealFlow, warp real images to create training pairs. However, these methods frequently introduce "ghosting" artifacts and blurriness, particularly at object boundaries. Furthermore, they often struggle to model independent object motion (IOM), assuming a static scene where only the camera moves. MPI-Flow proposes a 3D-aware pipeline to overcome these limitations.
Methodology
MPI-Flow moves beyond 2D pixel warping by adopting a Multiplane Image (MPI) representation. This approach constructs a stack of semi-transparent layers at varying depths from a single input image.
MPI Construction and Volume Rendering
From a single "seed" image, the system estimates depth and builds the MPI layers. Instead of flat 2D warping, the system manipulates these 3D planes based on virtual camera trajectories. Volume rendering is then used to project these planes back into a 2D frame. This process naturally handles occlusion and depth discontinuities, preventing the "holes" typically seen in 2D warping methods.
Independent Object Motion (IOM)
A key innovation of MPI-Flow is the explicit modeling of moving objects. Using instance segmentation, the system identifies and "cuts out" specific objects, such as vehicles or pedestrians.
By applying distinct motion vectors to these objects relative to the background, the pipeline generates training data that reflects the dynamic nature of real-world traffic and street scenes.
Depth-Aware Inpainting
When an object moves, it reveals previously occluded background areas. MPI-Flow employs a specialized depth-aware inpainting module to fill these regions realistically. This ensures that the synthesized frames maintain the visual consistency of real photographs rather than appearing as digital collages.
Experiments and Results
The researchers evaluated MPI-Flow using images from COCO and DAVIS to generate large-scale training sets, testing the resulting models on the Sintel and KITTI benchmarks.
Quantitative Performance
MPI-Flow sets a new state-of-the-art for unsupervised optical flow on the KITTI 2015 benchmark, achieving an Outlier Ratio (Fl) of 6.63%, showing strong performance against established methods like UFlow (9.05%). When used to fine-tune the popular RAFT model, MPI-Flow reached an End-Point Error (EPE) of 1.88 on KITTI 15, outperforming models trained on other real-image generators.
Qualitative Comparison
Visual analysis demonstrates that MPI-Flow produces significantly cleaner images with sharper boundaries compared to Depthstillation and RealFlow.
Discussion and Conclusion
While MPI-Flow represents a significant advancement, it currently treats objects as rigid bodies, failing to account for non-rigid motions like human gait. Additionally, the quality of the generated flow is dependent on the accuracy of the underlying depth estimation and segmentation models. Future work will focus on incorporating non-rigid motion and reducing the computational overhead associated with MPI volume rendering.
Despite these limitations, MPI-Flow demonstrates that a single high-quality photograph contains sufficient geometric information to synthesize complex, realistic motion sequences for training robust optical flow models.
The code is available on GitHub at Sharpiless/MPI-Flow.
Summary updated