A Deep Temporal Fusion Framework for Scene Flow Using a Learnable Motion Model and Occlusions

René Schuster, Christian Unger, Didier Stricker

TL;DR

This paper introduces Deep Temporal Fusion (DTF), a framework that upgrades standard two-frame scene flow estimators into high-performance multi-frame systems. By learning to "invert" motion from the past and fusing it with the future, the model significantly reduces errors in occluded areas while remaining orders of magnitude faster than traditional multi-frame methods.

What Problem Does This Paper Address?

In computer vision, Scene Flow is the holy grail of motion estimation: it describes the 3D position and 3D motion of every pixel in a scene. While 2D "optical flow" is common, scene flow is much harder because it requires understanding depth and motion simultaneously.

The biggest headache for scene flow is occlusion. When an object moves behind a parked car or exits the camera's field of view, a standard "dual-frame" (two-image) approach has no information to work with. Traditional "multi-frame" solutions try to fix this by looking at a sequence of images, but they usually rely on a "constant motion" assumption—basically guessing that objects move in straight lines at a steady speed. In the real world (like a busy intersection), this assumption often fails, and these older methods are notoriously slow, often taking minutes to process a single frame.

Methods

The authors propose a three-step pipeline that uses three consecutive stereo image pairs ($t-1, t, t+1$) to estimate motion at time $t$.

  1. Initial Estimation: They use an existing "off-the-shelf" scene flow estimator to get two raw predictions: a forward flow (where pixels go from $t \to t+1$) and a backward flow (where they came from at $t-1 \to t$).
  2. The Motion Inverter: Instead of assuming constant velocity, they built a dedicated sub-network to learn how to "flip" the backward motion into a forward prediction. This network uses spatial coordinates and depth information to account for the fact that 3D motion looks different when viewed from different points in time.
  3. Temporal Fusion: Finally, a fusion network takes the original forward estimate and the "inverted" backward estimate and blends them.

The "Secret Sauce": The fusion network predicts pixel-wise weights. If the forward estimate is likely occluded, the network assigns more weight to the inverted backward estimate. These weights naturally function as soft occlusion masks, providing a clear explanation of why the model chose one estimate over the other.

Datasets

The model was trained and evaluated using two primary benchmarks:

  • FlyingThings3D (FT3D): A massive synthetic dataset used for pre-training the motion inverter and fusion modules.
  • KITTI (2015): A gold-standard real-world dataset for autonomous driving. The authors used the KITTI training set for fine-tuning and the official online benchmark for final comparison against state-of-the-art methods.

Key Insights and Ideas

  • Learning > Heuristics: By replacing the "constant motion" math with a learnable neural network, the system handles complex, non-linear movements that traditional models miss.
  • Occlusion as a Byproduct: The framework doesn't need explicit labels for occlusions. It learns to identify them simply by trying to minimize the overall motion error during fusion.
  • Plug-and-Play: DTF isn't a standalone estimator; it's a wrapper. The authors showed it improves a variety of underlying models (like SENSE, DWARF, and PWOC-3D), making it highly versatile.
  • Efficiency: Because it relies on a feed-forward neural network rather than complex optimization loops, it is incredibly fast.

Key Results

  • Accuracy Boost: On the KITTI benchmark, the DTF framework reduced scene flow outliers by up to 30% in difficult occluded areas compared to the base dual-frame models.
  • Speed: The framework is a speed demon. While traditional multi-frame methods like PRSM take 300 seconds per frame, DTF (using PWOC-3D as a base) runs in just 0.38 seconds. This is a 2-to-5 order of magnitude speedup, bringing multi-frame scene flow much closer to real-time applications.
  • State-of-the-Art: At the time of publication, it achieved top-tier results on the KITTI leaderboard, particularly excelling at estimating the motion of foreground objects (like moving cars).

Limitations and Future Work

The authors identify the Motion Inverter as the current bottleneck. While it outperforms simple linear inversion, it still struggles with very complex 3D transformations.

Future work could involve:

  • End-to-End Integration: Training the base estimator and the fusion framework together from scratch, rather than as separate stages.
  • Geometric Constraints: Incorporating more explicit 3D geometric knowledge into the inverter to help it understand how camera ego-motion affects the scene.
  • Longer Sequences: Expanding the framework to look at more than three frames to provide even more temporal context.

Summary updated

Abstract

Motion estimation is one of the core challenges in computer vision. With traditional dual-frame approaches, occlusions and out-of-view motions are a limiting factor, especially in the context of environmental perception for vehicles due to the large (ego-) motion of objects. Our work pro-poses a novel data-driven approach for temporal fusion of scene flow estimates in a multi-frame setup to overcome the issue of occlusion. Contrary to most previous methods, we do not rely on a constant motion model, but instead learn a generic temporal relation of motion from data. In a second step, a neural network combines bi-directional scene flow estimates from a common reference frame, yielding a refined estimate and a natural byproduct of occlusion masks. This way, our approach provides a fast multi-frame extension for a variety of scene flow estimators, which outperforms the underlying dual-frame approaches.
BibTeX
@inproceedings{schuster2021deep,
  author = {René Schuster and Christian Unger and Didier Stricker},
  title = {{A Deep Temporal Fusion Framework for Scene Flow Using a Learnable Motion Model and Occlusions}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=vC5HaGsAO2z},
}

Loading related content...

Figure