Self-Superflow: Self-Supervised Scene Flow Prediction in Stereo Sequences

Katharina Bendig, René Schuster, Didier Stricker

TL;DR

Self-SuperFlow is a novel training framework that allows neural networks to learn 3D scene flow—the motion of objects in 3D space—using only unlabeled stereo video sequences. By using a clever "spatial flipping" trick to reason about occlusions and bidirectional motion, the method outperforms traditional supervised pre-training on synthetic data while converging nearly 30 times faster.

What Problem Does This Paper Address?

In the world of autonomous driving and robotics, "scene flow" is the holy grail of perception. Unlike optical flow, which only tracks 2D pixel movement on a screen, scene flow tracks the full 3D motion of every point in a scene.

The catch? Training deep learning models to predict scene flow usually requires massive amounts of "ground truth" data—perfectly labeled 3D motion vectors for every pixel. This is nearly impossible to capture in the real world. Consequently, researchers usually train models on synthetic (CGI) data and then try to "fine-tune" them for the real world. This creates a representation gap: models get really good at predicting motion in cartoons but struggle with the messy, inconsistent lighting and reflections of a real highway.

Methods

The authors propose Self-SuperFlow, a self-supervised approach that learns directly from raw, unlabeled stereo images. Their method relies on three main pillars:

  1. Photometric Reconstruction Loss: Instead of comparing predictions to a label, the model tries to "warp" one image onto another using its predicted flow. If the warped image matches the target image (measured via a Ternary Census Transform to handle lighting changes), the model knows its prediction was likely correct.
  2. Bidirectional Displacements via Spatial Flipping: To handle occlusions (areas that disappear or appear between frames), models need to calculate motion both forward and backward in time. Usually, this is computationally expensive. The authors introduced a "spatial flipping" strategy—flipping the images horizontally and temporally—to allow the network to predict inverse displacements without extra complex warping operations.
  3. Multi-Reference Consistency: Rather than just looking at the left camera at time t, the model calculates losses across all four available images (Left/Right at time t and t+1). This provides much tighter geometric constraints on what "correct" motion looks like.

The authors used PWOC-3D as their base architecture due to its efficiency, but the training framework itself is model-agnostic.

Datasets

  • KITTI Raw Dataset: Used for the primary self-supervised training (approx. 35,666 unlabeled samples).
  • KITTI 2015 Scene Flow: Used for validation, fine-tuning, and benchmarking against other state-of-the-art (SotA) methods.

Key Insights and Ideas

  • Real Data Trumps Synthetic Labels: The most striking finding is that training on real-world unlabeled data is significantly more effective than training on perfectly labeled synthetic data. The "domain gap" is a bigger hurdle than the lack of labels.
  • The Flipping "Hack": By flipping the input images, the authors found a way to get bidirectional flow "for free." This allows the model to perform a consistency check: if the forward flow and backward flow don't agree, the model assumes that area is occluded and ignores it during training.
  • Efficiency is King: Because the model is learning from data that matches its eventual task, it reaches peak performance in just 24 epochs, compared to the 760 epochs required for synthetic supervised pre-training.

Key Results

  • Accuracy: On the KITTI benchmark, Self-SuperFlow achieved a 14.73% outlier rate (SF-all), outperforming the supervised version of the same network (15.69%).
  • Outlier Reduction: Before any fine-tuning, the self-supervised approach had 20 percentage points fewer outliers than the synthetic supervised version on real-world validation data.
  • Convergence Speed: The method requires roughly 0.285M iterations to converge, whereas the supervised baseline required 7.84M iterations.
  • Monocular vs. Stereo: It drastically outperformed existing monocular (single camera) self-supervised methods, cutting outlier rates by more than half.

Limitations and Future Work

  • Foreground Challenges: The model still struggles slightly in foreground areas where motion is very fast and occlusions are frequent. In these specific regions, supervised models still hold a slight edge in "End-Point Error" (EPE).
  • Edge Blurring: The use of a "smoothness constraint" (to ensure neighboring pixels move similarly) can lead to slightly blurry edges around moving objects compared to supervised methods.
  • Future Directions: The authors suggest that further refining how the model handles occlusions in the foreground could bridge the remaining gap between self-supervised and fully supervised performance.

Summary updated

Abstract

In recent years, deep neural networks showed their exceeding capabilities in addressing many computer vision tasks including scene flow prediction. However, most of the advances are dependent on the availability of a vast amount of dense per pixel ground truth annotations, which are very difficult to obtain for real life scenarios. Therefore, synthetic data is often relied upon for supervision, resulting in a representation gap between the training and test data. Even though a great quantity of unlabeled real world data is available, there is a huge lack in self-supervised methods for scene flow prediction. Hence, we explore the extension of a self-supervised loss based on the Census transform and occlusion-aware bidirectional displacements for the problem of scene flow prediction. Regarding the KITTI scene flow benchmark, our method outperforms the corresponding supervised pre-training of the same network and shows improved generalization capabilities while achieving much faster convergence.
BibTeX
@inproceedings{bendig2022self,
  author = {Katharina Bendig and René Schuster and Didier Stricker},
  title = {{Self-Superflow: Self-Supervised Scene Flow Prediction in Stereo Sequences}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897832},
}

Loading related content...

Figure