Abstract
Motion estimation is one of the core challenges in computer vision. With traditional dual-frame approaches, occlusions and out-of-view motions are a limiting factor, especially in the context of environmental perception for vehicles due to the large (ego-) motion of objects. Our work proposes a novel data-driven approach for temporal fusion of scene flow estimates in a multi-frame setup to overcome the issue of occlusion. Contrary to most previous methods, we do not rely on a constant motion model, but instead learn a generic temporal relation of motion from data. In a second step, a neural network combines bi-directional scene flow estimates from a common reference frame, yielding a refined estimate and a natural byproduct of occlusion masks. This way, our approach provides a fast multi-frame extension for a variety of scene flow estimators, which outperforms the underlying dual-frame approaches.
Introduction
Scene flow estimation—the task of determining the 3D position and 3D motion of every pixel in a scene—is critical for autonomous navigation and environmental perception. However, standard "dual-frame" approaches, which rely on only two consecutive images, face significant challenges in the presence of occlusions. When objects move behind obstacles or exit the field of view, these models lack the necessary temporal context to maintain accurate estimates.
Traditional multi-frame solutions attempt to mitigate this by incorporating sequences of images, but they often rely on rigid "constant motion" heuristics. These assumptions frequently fail in dynamic real-world environments. Furthermore, these methods are often computationally expensive, requiring minutes of processing time per frame, which is impractical for real-time applications.
Methodology: Deep Temporal Fusion (DTF)
The proposed Deep Temporal Fusion (DTF) framework is a three-step pipeline designed to upgrade existing dual-frame scene flow estimators into high-performance multi-frame systems using three consecutive stereo image pairs ($t-1, t, t+1$).
- Initial Estimation: The framework utilizes an off-the-shelf scene flow estimator to generate two raw predictions: a forward flow ($t \to t+1$) and a backward flow ($t-1 \to t$).
- Motion Inverter: A dedicated sub-network learns to "invert" the backward motion into a forward prediction. Unlike linear mathematical inversion, this data-driven approach accounts for spatial coordinates and depth, learning how 3D motion manifests from different temporal perspectives.
- Temporal Fusion: A fusion network blends the original forward estimate with the inverted backward estimate. This network predicts pixel-wise weights, assigning higher importance to the inverted backward estimate in areas where the forward estimate is likely occluded. These weights naturally function as soft occlusion masks.
Evaluation and Results
The framework was trained on the synthetic FlyingThings3D (FT3D) dataset and fine-tuned on the real-world KITTI (2015) benchmark. By replacing heuristic-based motion models with a learnable neural network, the system effectively handles complex, non-linear movements.
Accuracy and Efficiency
On the KITTI benchmark, DTF reduced scene flow outliers by up to 30% in occluded regions compared to base dual-frame models. A key advantage of this approach is its efficiency. While traditional multi-frame methods like PRSM require approximately 300 seconds per frame, DTF (using PWOC-3D as a base) processes a frame in just 0.38 seconds. This represents a speedup of several orders of magnitude, making multi-frame scene flow viable for near real-time use.
Versatility
DTF is designed as a "plug-and-play" wrapper. Experiments demonstrated consistent performance improvements across various underlying estimators, including SENSE, DWARF, and PWOC-3D. At the time of its release, the framework achieved state-of-the-art results on the KITTI leaderboard, particularly in estimating the motion of foreground objects like moving vehicles.
Conclusion
The Deep Temporal Fusion framework provides a robust, efficient solution for multi-frame scene flow estimation. By learning temporal relations and occlusion patterns directly from data, it overcomes the limitations of traditional constant-motion models. Future work will focus on end-to-end integration of the base estimator and fusion framework, as well as incorporating explicit 3D geometric constraints to further refine the motion inversion process.
Summary updated