A Deep Temporal Fusion Framework for Scene Flow Using a Learnable Motion Model and Occlusions

René Schuster, Christian Unger, Didier Stricker

Abstract

Motion estimation is one of the core challenges in computer vision. With traditional dual-frame approaches, occlusions and out-of-view motions are a limiting factor, especially in the context of environmental perception for vehicles due to the large (ego-) motion of objects. Our work proposes a novel data-driven approach for temporal fusion of scene flow estimates in a multi-frame setup to overcome the issue of occlusion. Contrary to most previous methods, we do not rely on a constant motion model, but instead learn a generic temporal relation of motion from data. In a second step, a neural network combines bi-directional scene flow estimates from a common reference frame, yielding a refined estimate and a natural byproduct of occlusion masks. This way, our approach provides a fast multi-frame extension for a variety of scene flow estimators, which outperforms the underlying dual-frame approaches.

Introduction

Scene flow estimation—the task of determining the 3D position and 3D motion of every pixel in a scene—is critical for autonomous navigation and environmental perception. However, standard "dual-frame" approaches, which rely on only two consecutive images, face significant challenges in the presence of occlusions. When objects move behind obstacles or exit the field of view, these models lack the necessary temporal context to maintain accurate estimates.

Traditional multi-frame solutions attempt to mitigate this by incorporating sequences of images, but they often rely on rigid "constant motion" heuristics. These assumptions frequently fail in dynamic real-world environments. Furthermore, these methods are often computationally expensive, requiring minutes of processing time per frame, which is impractical for real-time applications.

Methodology: Deep Temporal Fusion (DTF)

The proposed Deep Temporal Fusion (DTF) framework is a three-step pipeline designed to upgrade existing dual-frame scene flow estimators into high-performance multi-frame systems using three consecutive stereo image pairs ($t-1, t, t+1$).

The Deep Temporal Fusion (DTF) pipeline architecture.
  1. Initial Estimation: The framework utilizes an off-the-shelf scene flow estimator to generate two raw predictions: a forward flow ($t \to t+1$) and a backward flow ($t-1 \to t$).
  2. Motion Inverter: A dedicated sub-network learns to "invert" the backward motion into a forward prediction. Unlike linear mathematical inversion, this data-driven approach accounts for spatial coordinates and depth, learning how 3D motion manifests from different temporal perspectives.
  3. Temporal Fusion: A fusion network blends the original forward estimate with the inverted backward estimate. This network predicts pixel-wise weights, assigning higher importance to the inverted backward estimate in areas where the forward estimate is likely occluded. These weights naturally function as soft occlusion masks.

Evaluation and Results

The framework was trained on the synthetic FlyingThings3D (FT3D) dataset and fine-tuned on the real-world KITTI (2015) benchmark. By replacing heuristic-based motion models with a learnable neural network, the system effectively handles complex, non-linear movements.

Qualitative comparison and fusion weights on the KITTI dataset.

Accuracy and Efficiency

On the KITTI benchmark, DTF reduced scene flow outliers by up to 30% in occluded regions compared to base dual-frame models. A key advantage of this approach is its efficiency. While traditional multi-frame methods like PRSM require approximately 300 seconds per frame, DTF (using PWOC-3D as a base) processes a frame in just 0.38 seconds. This represents a speedup of several orders of magnitude, making multi-frame scene flow viable for near real-time use.

Versatility

DTF is designed as a "plug-and-play" wrapper. Experiments demonstrated consistent performance improvements across various underlying estimators, including SENSE, DWARF, and PWOC-3D. At the time of its release, the framework achieved state-of-the-art results on the KITTI leaderboard, particularly in estimating the motion of foreground objects like moving vehicles.

Conclusion

The Deep Temporal Fusion framework provides a robust, efficient solution for multi-frame scene flow estimation. By learning temporal relations and occlusion patterns directly from data, it overcomes the limitations of traditional constant-motion models. Future work will focus on end-to-end integration of the base estimator and fusion framework, as well as incorporating explicit 3D geometric constraints to further refine the motion inversion process.

Summary updated

Abstract

Motion estimation is one of the core challenges in computer vision. With traditional dual-frame approaches, occlusions and out-of-view motions are a limiting factor, especially in the context of environmental perception for vehicles due to the large (ego-) motion of objects. Our work proposes a novel data-driven approach for temporal fusion of scene flow estimates in a multi-frame setup to overcome the issue of occlusion. Contrary to most previous methods, we do not rely on a constant motion model, but instead learn a generic temporal relation of motion from data. In a second step, a neural network combines bi-directional scene flow estimates from a common reference frame, yielding a refined estimate and a natural byproduct of occlusion masks. This way, our approach provides a fast multi-frame extension for a variety of scene flow estimators, which outperforms the underlying dual-frame approaches.
BibTeX
@inproceedings{schuster2021deep,
  author = {René Schuster and Christian Unger and Didier Stricker},
  title = {{A Deep Temporal Fusion Framework for Scene Flow Using a Learnable Motion Model and Occlusions}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=vC5HaGsAO2z},
}
Figure