Learning End-To-End Scene Flow by Distilling Single Tasks Knowledge

Filippo Aleotti, Matteo Poggi, Fabio Tosi, Stefano Mattoccia

Abstract

Scene flow is a challenging task aimed at jointly estimating the 3D structure and motion of the sensed environment. Although deep learning solutions achieve outstanding performance in terms of accuracy, these approaches divide the whole problem into standalone tasks (stereo and optical flow) addressing them with independent networks. Such a strategy dramatically increases the complexity of the training procedure and requires power-hungry GPUs to infer scene flow barely at 1 FPS. Conversely, we propose DWARF, a novel and lightweight architecture able to infer full scene flow jointly reasoning about depth and optical flow easily and elegantly trainable end-to-end from scratch. Moreover, since ground truth images for full scene flow are scarce, we propose to leverage on the knowledge learned by networks specialized in stereo or flow, for which much more data are available, to distill proxy annotations. Exhaustive experiments show that i) DWARF runs at about 10 FPS on a single high-end GPU and about 1 FPS on NVIDIA Jetson TX2 embedded at KITTI resolution, with moderate drop in accuracy compared to 10x deeper models, ii) learning from many distilled samples is more effective than from the few, annotated ones available.

Introduction

Scene flow estimation represents the dense 3D motion field of an environment, providing the 3D position of every pixel and its subsequent 3D displacement between consecutive frames. This task typically requires four input images: a stereo pair at time $t_1$ and another at time $t_2$.

Traditional deep learning approaches often decompose this into two independent sub-problems: stereo depth (disparity) estimation and 2D motion (optical flow) estimation. By stacking massive, specialized networks for each task, these methods achieve high accuracy but suffer from significant computational bloat, often running at only 1 FPS on high-end GPUs. Furthermore, this decoupled approach ignores the physical constraints linking depth, motion, and camera ego-motion. Finally, while datasets for individual tasks are abundant, real-world datasets with full scene flow ground truth are extremely scarce, limiting the scalability of supervised learning.

The DWARF Architecture

To address these challenges, we introduce DWARF (Disparity, Warping, and Flow), a lightweight, end-to-end architecture designed for joint inference. DWARF utilizes a coarse-to-fine pyramidal structure that simultaneously estimates disparity, optical flow, and disparity change.

The DWARF architecture features a shared pyramidal feature extractor and a joint decoder that reasons across multiple scales to estimate disparity and flow.

A key innovation in DWARF is the integration of joint reasoning within the decoder. While standard networks use 1D correlation for stereo and 2D correlation for flow, DWARF employs a custom 3D correlation layer at the coarsest level. This layer correlates 1D stereo matching cost curves across time within a 2D spatial search window, explicitly tracking how a pixel's depth signature evolves as it moves. This provides a strong inductive bias, allowing a much smaller network to capture complex physical relationships that larger, stacked models might miss.

Knowledge Distillation from Expert Models

To overcome the lack of annotated scene flow data, we employ a "Teacher-Student" distillation strategy. We leverage massive, state-of-the-art "expert" networks trained specifically for single tasks (e.g., FlowNet-CSS for flow and DispNet-CSS for disparity) to generate proxy scene flow labels on thousands of unannotated real-world images.

Knowledge distillation pipeline: large expert models provide proxy annotations to train the lightweight student model on unannotated data.

By training DWARF on this large-scale "Distilled-KITTI" dataset (comprising 4,000 images), the model learns from a diverse set of real-world scenarios. Our findings indicate that training on a large volume of slightly noisy proxy labels is more effective than relying solely on the very limited (approx. 200 images) perfect ground truth available in datasets like KITTI 2015.

Experimental Results

DWARF demonstrates a remarkable balance between efficiency and accuracy. It contains only 19.6 million parameters, roughly 6x-10x fewer than competing state-of-the-art models.

  • Performance: DWARF achieves approximately 10 FPS on an NVIDIA GTX 1080Ti and 1 FPS on an embedded NVIDIA Jetson TX2 (drawing <10W). This makes real-time scene flow feasible for edge devices and robotics.
  • Accuracy: On the KITTI 2015 benchmark, DWARF outperformed other lightweight models like PWOC-3D by nearly 3% on the F1-All metric, maintaining competitive rankings despite its significantly smaller footprint.
  • Generalization: Qualitative tests on the WeanHall indoor dataset confirm that the model generalizes well to environments outside its training distribution.

Conclusion

DWARF proves that joint reasoning and knowledge distillation can enable high-quality scene flow estimation on resource-constrained hardware. By abandoning the stacked-network paradigm in favor of a unified, lightweight architecture, we pave the way for real-time 3D perception in autonomous systems. Future work will explore self-supervised refinement techniques to allow the model to adapt continuously to new environments without requiring any offline distillation or ground truth.

Summary updated

Abstract

Scene flow is a challenging task aimed at jointly estimating the 3D structure and motion of the sensed environment. Although deep learning solutions achieve outstanding performance in terms of accuracy, these approaches divide the whole problem into standalone tasks (stereo and optical flow) addressing them with independent networks. Such a strategy dramatically increases the complexity of the training procedure and requires power-hungry GPUs to infer scene flow barely at 1 FPS. Conversely, we propose DWARF, a novel and lightweight architecture able to infer full scene flow jointly reasoning about depth and optical flow easily and elegantly trainable end-to-end from scratch. Moreover, since ground truth images for full scene flow are scarce, we propose to leverage on the knowledge learned by networks specialized in stereo or flow, for which much more data are available, to distill proxy annotations. Exhaustive experiments show that i) DWARF runs at about 10 FPS on a single high-end GPU and about 1 FPS on NVIDIA Jetson TX2 embedded at KITTI resolution, with moderate drop in accuracy compared to 10x deeper models, ii) learning from many distilled samples is more effective than from the few, annotated ones available. Code available at: https://github.com/FilippoAleotti/Dwarf-Tensorflow
BibTeX
@inproceedings{aleotti2020learning,
  author = {Filippo Aleotti and Matteo Poggi and Fabio Tosi and Stefano Mattoccia},
  title = {{Learning End-To-End Scene Flow by Distilling Single Tasks Knowledge}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6613},
}
Figure