Learning Residual Flow as Dynamic Motion from Stereo Videos

Seokju Lee, Sunghoon Im, Stephen Lin, In So Kweon

TL;DR

This paper introduces an unsupervised learning framework that decomposes 3D scene flow into two parts: stationary background motion caused by the camera and the independent "residual" motion of dynamic objects. By leveraging stereo video and a multi-phase training strategy, the authors achieve state-of-the-art results in both optical flow and visual odometry without requiring labeled ground truth.

What Problem Does This Paper Address?

For a robot or autonomous vehicle to navigate safely, it must distinguish between its own movement (ego-motion) and the movement of other objects in the environment (dynamic motion).

While deep learning has made strides in estimating depth and ego-motion from monocular video, these methods often struggle in busy urban environments. In a monocular setup, there is an inherent ambiguity: is that car getting larger because I am moving toward it, or because it is moving toward me? Furthermore, standard unsupervised methods often treat moving objects as "outliers" to be masked out, rather than entities to be explicitly understood. This paper seeks to jointly reason about camera motion, depth, and the 3D motion of dynamic objects to solve this ambiguity.

Methods

The authors propose a system composed of three cooperating neural networks:

  1. DepthNet: Uses a stereo matching network (specifically a pre-trained PSMNet) to estimate highly accurate depth maps. Using stereo instead of monocular images removes scale ambiguity.
  2. PoseNet: Predicts the 6-DoF (Degrees of Freedom) camera motion between consecutive frames.
  3. ResFlowNet (The Core Innovation): Instead of predicting total optical flow directly, this network predicts Residual Flow ($F^{res}$).

The "Total Flow" is formulated as the sum of Rigid Flow (motion explained by camera movement and scene depth) and Residual Flow (motion caused by moving objects). $$F^{tot} = F^{rig} + F^{res}$$

Key Technical Innovations:

  • Multi-Phase Training: The model is trained in three stages. First, PoseNet is trained on static scenes. Second, ResFlowNet is trained to capture what PoseNet missed. Finally, they are trained jointly, where the total flow helps refine the camera pose estimation by providing a better reconstruction of the target image.
  • Depth-Aware Smoothness Loss: Traditional flow models use image gradients to smooth the flow. However, image gradients are noisy (e.g., road markings look like edges but shouldn't break flow). This paper uses second-order depth gradients to regularize the flow, ensuring it remains smooth across flat surfaces like roads while respecting actual geometric boundaries.
  • Two-Stage Validity Mask: To handle occlusions (areas visible in one frame but not the next), the system uses a mask that filters out pixels where the forward and backward flow don't match, preventing the network from learning from "garbage" data in occluded regions.

Datasets

The model was trained and evaluated using the KITTI dataset, a standard benchmark for autonomous driving:

  • KITTI Raw Stereo Videos: Used for unsupervised training (approx. 20,000 image pairs).
  • KITTI 2015 Scene Flow: Used to evaluate optical flow performance.
  • KITTI Odometry: Used to evaluate camera pose estimation (Visual Odometry).

Key Insights and Ideas

  • Residuals as Dynamics: By defining dynamic motion as a residual of the rigid geometric transformation, the model naturally decouples the physics of the camera from the physics of the objects.
  • Stereo is a "Cheat Code" for Depth: Relying on stereo matching provides a much more stable foundation for ego-motion than monocular depth estimation, which often collapses in complex scenes.
  • The "Aha" Moment in Training: The authors found that training the camera pose network using the total flow (including the dynamic parts) actually improved ego-motion accuracy. This suggests that a model which understands why a pixel moved (even if it moved "wrongly" relative to the camera) becomes better at ignoring that pixel when calculating its own path.

Key Results

The proposed method outperformed existing unsupervised state-of-the-art algorithms across multiple tasks:

  • Optical Flow: On the KITTI 2015 benchmark, the model achieved an Average Endpoint Error (EPE) of 8.74, significantly better than other unsupervised methods like GeoNet (10.81) and competitive with some supervised methods.
  • Visual Odometry: The model achieved lower Absolute Trajectory Error (ATE) on KITTI sequences 09 and 10 compared to popular baselines like SfM-Learner and Vid2Depth.
  • 3D Motion Prediction: The system successfully back-projects 2D flow into 3D space, allowing it to estimate the actual metric velocity (m/s) of moving cars in the scene.

Limitations and Future Work

  • Instance Dependence: While the model predicts 3D motion, it currently relies on an external "off-the-shelf" instance segmentation network (PANet) to group pixels into specific objects.
  • Generic Rigid Bodies: The current framework is optimized for the types of motion seen in urban driving. The authors suggest that future work will focus on generalizing this to any type of rigid body motion and improving dynamic region segmentation.
  • Photometric Limitations: Like all unsupervised methods based on "warping" images, it can still struggle with extreme lighting changes or perfectly featureless surfaces where the photometric loss becomes uninformative.

Summary updated

Abstract

We present a method for decomposing the 3D scene flow observed from a moving stereo rig into stationary scene elements and dynamic object motion. Our unsupervised learning framework jointly reasons about the camera motion, optical flow, and 3D motion of moving objects. Three cooperating networks predict stereo matching, camera motion, and residual flow, which represents the flow component due to object motion and not from camera motion. Based on rigid projective geometry, the estimated stereo depth is used to guide the camera motion estimation, and the depth and camera motion are used to guide the residual flow estimation. We also explicitly estimate the 3D scene flow of dynamic objects based on the residual flow and scene depth. Experiments on the KITTI dataset demonstrate the effectiveness of our approach and show that our method outperforms other state-of-the-art algorithms on the optical flow and visual odometry tasks.
BibTeX
@inproceedings{lee2019learning,
  author = {Seokju Lee and Sunghoon Im and Stephen Lin and In So Kweon},
  title = {{Learning Residual Flow as Dynamic Motion from Stereo Videos}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=L6DxpPVQp2},
}

Loading related content...

Figure