SceneFlowFields: Dense Interpolation of Sparse Scene Flow Correspondences

René Schuster, Oliver Wasenmüller, Georg Kuschk, Christian Bailer, Didier Stricker

Abstract

While most scene flow methods use either variational optimization or a strong rigid motion assumption, we show for the first time that scene flow can also be estimated by dense interpolation of sparse matches. To this end, we find sparse matches across two stereo image pairs that are detected without any prior regularization and perform dense interpolation preserving geometric and motion boundaries by using edge information. A few iterations of variational energy minimization are performed to refine our results, which are thoroughly evaluated on the KITTI benchmark and additionally compared to state-of-the-art on MPI Sintel. For application in an automotive context, we further show that an optional ego-motion model helps to boost performance and blends smoothly into our approach to produce a segmentation of the scene into static and dynamic parts.

Introduction

Scene flow represents the 3D motion of every point in a scene between two consecutive time steps. It is a critical component for autonomous driving and robotic navigation, yet it remains computationally challenging. Traditional approaches typically fall into two categories: complex variational optimization, which is slow and struggles with large displacements, or rigid motion assumptions, which often fail on deformable objects or complex dynamic environments.

SceneFlowFields introduces a novel paradigm by estimating dense 3D scene flow through the interpolation of sparse, high-quality matches. By moving away from heavy global regularization and utilizing edge-preserving interpolation with a semantic-aware boundary detector, this method achieves state-of-the-art accuracy on the KITTI benchmark while remaining significantly faster than traditional variational methods.

Methodology

The SceneFlowFields pipeline is structured into several primary stages designed to handle large displacements and maintain boundary sharpness.

The SceneFlowFields pipeline flowchart, illustrating the stages from initialization to adjusted scene flow.
  1. Sparse Matching: The process begins by finding correspondences across four images (stereo pairs at $t_0$ and $t_1$). A multi-scale propagation technique searches a 4D space comprising 2D optical flow $(u, v)$ and two disparities $(d_0, d_1)$. This joint optimization exploits stereo redundancies for better coherence.
  2. Consistency Filtering: A forward-backward consistency check is applied to discard unreliable matches, ensuring only high-quality "seeds" proceed to interpolation.
  3. Edge-Preserving Interpolation: This core innovation uses the sparse matches to fill the dense field. To prevent motion "bleeding" across object boundaries, the authors employ a Structured Edge Detector trained on semantic labels (e.g., "car" vs. "road"). This ensures interpolation respects actual geometric and motion boundaries rather than just color gradients.
  4. Variational Refinement: A lightweight variational energy minimization step is performed to polish the interpolated field, enhancing sub-pixel accuracy.
  5. Optional Ego-Motion: For automotive applications, an optional module estimates camera motion to better model the static background, facilitating a segmentation of the scene into static and dynamic components.

Evaluation and Results

The method was evaluated on the KITTI Scene Flow Benchmark and the MPI Sintel dataset.

Visual results from the SceneFlowFields pipeline: a) original image, b) semantic edge map, c) estimated scene flow, and d) 3D point cloud visualization.

Performance and Efficiency

On the KITTI benchmark, SceneFlowFields ranked 3rd among all dual-frame methods at the time of publication. Crucially, it demonstrated a significant speed advantage: while top-performing variational methods required approximately 3000 seconds per frame, SceneFlowFields completed the task in just 65 seconds.

Generalization

The method's strong performance on the synthetic MPI Sintel dataset demonstrates its robustness to non-rigid motion and large displacements, proving its utility beyond automotive-specific scenarios.

Conclusion

SceneFlowFields demonstrates that the "matching + interpolation" strategy is highly effective for 4D scene flow estimation. By prioritizing semantic boundaries over simple color edges, the method maintains high accuracy at object interfaces. While 65 seconds per frame is not yet real-time, it represents a massive leap in efficiency over traditional variational approaches. Future work will focus on improving ego-motion robustness and incorporating temporal consistency across multiple frames to further refine the 3D motion field.

Summary updated

Abstract

While most scene flow methods use either variational optimization or a strong rigid motion assumption, we show for the first time that scene flow can also be estimated by dense interpolation of sparse matches. To this end, we find sparse matches across two stereo image pairs that are detected without any prior regularization and perform dense interpolation preserving geometric and motion boundaries by using edge information. A few iterations of variational energy minimization are performed to refine our results, which are thoroughly evaluated on the KITTI benchmark and additionally compared to state-of-the-art on MPI Sintel. For application in an automotive context, we further show that an optional ego-motion model helps to boost performance and blends smoothly into our approach to produce a segmentation of the scene into static and dynamic parts.
BibTeX
@inproceedings{schuster2018sceneflowfields,
  author = {René Schuster and Oliver Wasenmüller and Georg Kuschk and Christian Bailer and Didier Stricker},
  title = {{SceneFlowFields: Dense Interpolation of Sparse Scene Flow Correspondences}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=8AvKlaVo7R},
}
Figure