Self-SuperFlow: Self-supervised Scene Flow Prediction in Stereo Sequences

Katharina Bendig, René Schuster, Didier Stricker

Abstract

In recent years, deep neural networks have shown exceeding capabilities in addressing computer vision tasks, including scene flow prediction. However, most advances depend on vast amounts of dense per-pixel ground truth annotations, which are difficult to obtain for real-life scenarios. Consequently, synthetic data is often used for supervision, resulting in a representation gap between training and test data. While unlabeled real-world data is abundant, there is a lack of self-supervised methods for scene flow. We explore a self-supervised loss based on the Census transform and occlusion-aware bidirectional displacements. On the KITTI scene flow benchmark, our method outperforms supervised pre-training of the same network, shows improved generalization, and achieves significantly faster convergence.

Introduction

In autonomous driving and robotics, "scene flow" represents the holy grail of perception. Unlike optical flow, which tracks 2D pixel movement, scene flow tracks the full 3D motion of every point in a scene. Predicting this motion is critical for safety-critical tasks like obstacle avoidance and path planning.

Traditionally, training models to predict scene flow requires massive datasets with "ground truth" labels—perfectly accurate 3D motion vectors for every pixel. Because these labels are nearly impossible to capture in the real world, researchers rely on synthetic (CGI) data. This creates a representation gap: models trained on synthetic data often struggle with the messy lighting, reflections, and sensor noise of real-world environments.

Self-SuperFlow addresses this by learning directly from raw, unlabeled stereo video sequences, bridging the gap between simulation and reality.

The Self-SuperFlow Framework

Self-SuperFlow is a model-agnostic training framework that allows neural networks to learn 3D motion without human-provided labels. The method relies on three primary technical pillars:

1. Photometric Reconstruction Loss

Instead of comparing a prediction to a label, the model uses its predicted flow to "warp" one image onto another. If the prediction is accurate, the warped image should match the target image. To ensure robustness against lighting changes and shadows, the authors employ a Ternary Census Transform, which focuses on local pixel structures rather than absolute intensity values.

2. Bidirectional Displacements via Spatial Flipping

Handling occlusions—areas that appear or disappear between frames—is a major challenge in motion estimation. To identify these regions, models must calculate motion both forward and backward in time. Typically, this is computationally expensive. Self-SuperFlow introduces a "spatial flipping" strategy: by flipping images horizontally and temporally, the network can predict inverse displacements without requiring additional complex warping operations or redundant passes.

3. Multi-Reference Consistency

By utilizing all four images available in a stereo sequence (Left and Right cameras at times t and t+1), the model enforces tight geometric constraints. This multi-view consistency ensures that the predicted 3D motion is physically plausible across both the temporal (motion) and spatial (stereo) dimensions.

Experimental Results

The authors implemented their framework using the PWOC-3D architecture and evaluated it on the KITTI Scene Flow benchmark.

Performance and Accuracy

Self-SuperFlow demonstrated that real-world data is often more valuable than synthetic labels. On the KITTI benchmark, the method achieved a 14.73% outlier rate (SF-all), actually outperforming the supervised version of the same network (15.69%). Before any fine-tuning, the self-supervised approach showed 20 percentage points fewer outliers than the synthetic supervised version when tested on real-world data.

Efficiency and Convergence

One of the most significant advantages of the self-supervised approach is the speed of training. Because the model learns from data that matches the target domain, it reaches peak performance in just 24 epochs (approximately 0.285M iterations). In contrast, the supervised baseline required 760 epochs (7.84M iterations) to converge—a nearly 30x difference in training efficiency.

Discussion and Limitations

While Self-SuperFlow marks a significant step forward, some challenges remain. The model currently struggles in foreground areas characterized by very fast motion and frequent occlusions. In these specific regions, supervised models still maintain a slight edge in End-Point Error (EPE). Additionally, the use of smoothness constraints—intended to ensure neighboring pixels move consistently—can occasionally lead to blurred edges around moving objects.

Conclusion

Self-SuperFlow demonstrates that self-supervision is not just a viable alternative to supervised learning for scene flow, but in many cases, a superior one. By leveraging the abundance of unlabeled real-world stereo data and employing clever geometric tricks like spatial flipping, the framework achieves state-of-the-art results with a fraction of the training time required by traditional methods. This paves the way for more scalable and robust perception systems in autonomous vehicles.

Summary updated

Abstract

In recent years, deep neural networks showed their exceeding capabilities in addressing many computer vision tasks including scene flow prediction. However, most of the advances are dependent on the availability of a vast amount of dense per pixel ground truth annotations, which are very difficult to obtain for real life scenarios. Therefore, synthetic data is often relied upon for supervision, resulting in a representation gap between the training and test data. Even though a great quantity of unlabeled real world data is available, there is a huge lack in self-supervised methods for scene flow prediction. Hence, we explore the extension of a self-supervised loss based on the Census transform and occlusion-aware bidirectional displacements for the problem of scene flow prediction. Regarding the KITTI scene flow benchmark, our method outperforms the corresponding supervised pre-training of the same network and shows improved generalization capabilities while achieving much faster convergence.
BibTeX
@inproceedings{bendig2022self,
  author = {Katharina Bendig and René Schuster and Didier Stricker},
  title = {{Self-SuperFlow: Self-supervised Scene Flow Prediction in Stereo Sequences}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897832},
}
Figure