Abstract
We present a method for decomposing the 3D scene flow observed from a moving stereo rig into stationary scene elements and dynamic object motion. Our unsupervised learning framework jointly reasons about the camera motion, optical flow, and 3D motion of moving objects. Three cooperating networks predict stereo matching, camera motion, and residual flow, which represents the flow component due to object motion and not from camera motion. Based on rigid projective geometry, the estimated stereo depth is used to guide the camera motion estimation, and the depth and camera motion are used to guide the residual flow estimation. We also explicitly estimate the 3D scene flow of dynamic objects based on the residual flow and scene depth. Experiments on the KITTI dataset demonstrate the effectiveness of our approach and show that our method outperforms other state-of-the-art algorithms on the optical flow and visual odometry tasks.
Introduction
For autonomous vehicles to navigate safely, they must distinguish between their own movement (ego-motion) and the movement of other objects in the environment. While deep learning has improved depth and ego-motion estimation from monocular video, these methods often struggle in busy urban environments due to scale ambiguity—for instance, determining if a car is growing larger because the observer is moving or because the car itself is approaching.
Standard unsupervised methods often treat moving objects as outliers to be masked out. This paper proposes a framework that jointly reasons about camera motion, depth, and the 3D motion of dynamic objects to solve this ambiguity by explicitly modeling the "residual" motion of dynamic entities.
Methodology
The proposed system utilizes three cooperating neural networks to decompose 3D scene flow into stationary background motion and independent object motion:
- DepthNet: A stereo matching network (based on a pre-trained PSMNet) that estimates accurate depth maps, leveraging stereo vision to eliminate scale ambiguity.
- PoseNet: Predicts the 6-DoF camera motion between consecutive frames.
- ResFlowNet: The core innovation, which predicts Residual Flow ($F^{res}$).
In this framework, Total Flow ($F^{tot}$) is formulated as the sum of Rigid Flow ($F^{rig}$)—motion explained by camera movement and scene depth—and Residual Flow—motion caused by moving objects: $$F^{tot} = F^{rig} + F^{res}$$
Technical Innovations
- Multi-Phase Training: The model is trained in stages: first PoseNet on static scenes, then ResFlowNet to capture remaining motion, and finally a joint training phase where total flow refines camera pose estimation.
- Depth-Aware Smoothness Loss: Instead of noisy image gradients, the system uses second-order depth gradients to regularize flow, ensuring smoothness across flat surfaces like roads while respecting geometric boundaries.
- Validity Masking: A two-stage mask filters out occluded regions where forward and backward flow do not match, preventing the network from learning from unreliable data.
Results and Discussion
The model was evaluated on the KITTI dataset, achieving state-of-the-art results for unsupervised methods:
- Optical Flow: On the KITTI 2015 benchmark, the model achieved an Average Endpoint Error (EPE) of 8.74, outperforming previous unsupervised methods like GeoNet (10.81).
- Visual Odometry: The system achieved lower Absolute Trajectory Error (ATE) on KITTI sequences compared to baselines like SfM-Learner.
- 3D Motion Prediction: By back-projecting 2D flow into 3D space, the system estimates the metric velocity (m/s) of moving objects.
Conclusion
By defining dynamic motion as a residual of rigid geometric transformations, this framework successfully decouples camera physics from object physics. While the current model relies on external instance segmentation for grouping pixels into objects, it provides a robust foundation for unsupervised scene flow estimation. Future work will focus on improving dynamic region segmentation and generalizing the framework to a wider variety of rigid body motions.
Summary updated