Abstract
Estimating 3D scene flow—the dense 3D motion of every point in a scene—is a fundamental challenge in computer vision, typically hindered by high computational costs and a lack of labeled data. We present DWARF (Disparity, Warping, and Flow), a lightweight, end-to-end architecture that achieves 10 FPS on standard hardware. By utilizing a novel 3D correlation layer and knowledge distillation from specialized expert models, DWARF overcomes data scarcity and provides a real-time solution suitable for embedded systems in autonomous driving and robotics.
Introduction
Scene flow is often considered the "holy grail" of low-level vision because it requires solving for both depth (stereo matching) and 2D motion (optical flow) simultaneously. Existing high-performance models often stack massive, independent networks, leading to "power-hungry" architectures that fail to meet real-time requirements. Furthermore, while datasets for individual tasks like stereo or flow are abundant, ground-truth labels for full 3D scene flow are extremely scarce, making it difficult to train unified models from scratch.
Methodology
DWARF is designed as a unified, coarse-to-fine pyramidal architecture that is both elegant and lightweight. It introduces three primary innovations to improve efficiency and accuracy:
3D Correlation Layer
Unlike standard models that use 1D correlations for stereo and 2D for flow, DWARF employs a 3D correlation layer. This layer computes correlations over correlation volumes, allowing the network to explicitly reason about how depth changes over time. This serves as a geometric bridge between spatial structure and temporal motion, capturing relationships without requiring a deeper, more complex network.
Knowledge Distillation
To address the shortage of labeled scene flow data, the authors adopt a "teacher-student" strategy. Large, state-of-the-art "expert" models generate proxy labels on 4,000 unlabeled KITTI images. DWARF is then trained to mimic these experts, effectively inheriting their performance while maintaining a much smaller parameter footprint. This approach provided a 20x increase in usable training data for the KITTI environment.
Joint Estimation
DWARF utilizes a shared feature encoder and a single "Estimator" block at each pyramid level. This block features three specialized heads to predict disparity, optical flow, and disparity change simultaneously. By reasoning about these inherently dependent tasks jointly, the model reduces the redundant computations found in stacked architectures.
Key Results
The DWARF architecture demonstrates significant improvements in both speed and efficiency:
- Real-Time Performance: The model runs at 10 FPS on an NVIDIA Titan 1080Ti and 1 FPS on the embedded NVIDIA Jetson TX2 module.
- Model Efficiency: DWARF is approximately 10x smaller in terms of parameters compared to leading models like Ilg et al. (2018) while remaining competitive on the KITTI benchmark.
- Accuracy: On KITTI 2015, DWARF significantly outperformed other fast models, reducing the Outliers error rate by nearly 3% across several metrics.
- Generalization: Testing on the indoor WeanHall dataset confirmed that the model generalizes well beyond its driving-centric training data to indoor robotic environments.
Discussion and Limitations
While DWARF is much faster, there is still a moderate drop in accuracy compared to the heaviest, most computationally expensive models. It currently ranks 15th on the KITTI leaderboard, suggesting room for improvement in fine-grained estimation. Additionally, the labels provided by the "experts" are noisy, which likely places a ceiling on the student model's learning potential. Future work will explore self-adaptive versions of DWARF that can continue refining their estimations in an online manner as they encounter new environments.
Summary updated