Learning Optical Flow, Depth, and Scene Flow Without Real-World Labels

Vitor Guizilini, Kuan-Hui Lee, Rares Ambrus, Adrien Gaidon

Abstract

Self-supervised learning of 3D scene structure and motion from monocular video typically relies on the "static world assumption," which fails in dynamic environments containing moving objects. We present DRAFT (Depth-RAFT), a novel architecture that jointly learns depth, optical flow, and scene flow without requiring real-world labels or stereo pairs during training. By bootstrapping from 2D optical flow and utilizing an iterative GRU-based refinement process trained on a mixture of synthetic and real-world data, DRAFT achieves state-of-the-art performance on the KITTI benchmark. Our approach demonstrates that explicitly modeling scene flow improves depth accuracy by effectively handling dynamic objects that are typically treated as outliers.

Introduction

In autonomous driving, understanding 3D structure and motion is critical. Traditional supervised methods require massive hand-labeled datasets or expensive LiDAR sensors. While self-supervised learning offers a scalable alternative by using raw video, most existing models rely on "view synthesis" and the static world assumption. This assumption posits that pixel changes between frames are solely due to camera motion and scene depth. However, real-world roads are dynamic; independently moving objects like cars and pedestrians break the geometric constraints of static-world models, leading to ill-posed problems where infinite combinations of depth and motion could explain the observed video.

Methodology

DRAFT (Depth-RAFT) addresses these challenges by building upon the RAFT architecture to learn depth and 3D scene flow iteratively. The pipeline consists of several key stages:

  1. Optical Flow Bootstrapping: The model first estimates 2D optical flow. Since optical flow is a 2D displacement task, it is inherently robust to the static world assumption.
  2. Triangulation Initialization: This estimated flow is used to "triangulate" an initial depth map. While noisy for moving objects, it provides a strong geometric starting point for static regions.
  3. Iterative Refinement: A GRU-based recurrent update operator iteratively refines the depth and 3D scene flow predictions.
  4. Mixed-Batch Training: To bridge the gap between simulation and reality, the model is trained on a mix of synthetic data (VKITTI2 and Parallel Domain) and unlabeled real-world video (KITTI). Synthetic data provides perfect geometric priors and metric scale, while real-world data ensures generalization.
  5. Geometric Consistency: The model is constrained by losses that ensure predicted 3D scene flow and depth, when projected back to 2D, remain consistent with the predicted optical flow.
The DRAFT architecture overview, showing the iterative refinement of depth and scene flow from optical flow initialization.

Results and Discussion

DRAFT establishes a new state-of-the-art for self-supervised monocular methods on the KITTI benchmark across depth, optical flow, and scene flow tasks.

  • Depth Estimation: DRAFT achieved a lower Sq.Rel (0.647) and RMSE (3.991) compared to previous leading methods like ManyDepth. This improvement is largely attributed to better handling of dynamic objects.
  • Optical Flow: The model significantly outperformed the RAFT baseline in a self-supervised setting, reducing the End-Point Error (EPE) to 2.54.
  • Scene Flow: Despite being monocular, DRAFT's scene flow results (SF1-all: 37.58) are competitive with older stereo-based methods, proving the efficacy of joint learning.
Example of a predicted depth map generated by DRAFT, demonstrating its ability to capture complex scene geometry.

A key insight from this work is that scene flow and depth learning are complementary. Rather than destabilizing depth estimation, explicit scene flow modeling allows the network to account for moving objects, which improves overall depth accuracy. Furthermore, the authors demonstrate that simple mixed-batch training is an effective alternative to complex domain adaptation for transferring scale from simulation to the real world.

Conclusion

DRAFT demonstrates that joint learning of depth and motion can overcome the limitations of the static world assumption in self-supervised monocular vision. By leveraging synthetic data for geometric priors and iterative refinement for precision, DRAFT provides a robust framework for understanding dynamic 3D environments without the need for expensive real-world labels. Future work may focus on reducing the computational cost of the iterative GRU updates and further narrowing the performance gap between monocular and stereo-based scene flow models.

Summary updated

BibTeX
@article{guizilini2022learning,
  author = {Vitor Guizilini and Kuan-Hui Lee and Rares Ambrus and Adrien Gaidon},
  title = {{Learning Optical Flow, Depth, and Scene Flow Without Real-World Labels}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  doi = {10.1109/lra.2022.3145057},
}
Figure