Data-free Dense Depth Distillation

Junjie Hu, Chenyou Fan, Mete Ozay, Hualie Jiang, Tin Lun Lam

Abstract

This paper presents a novel framework for "data-free" knowledge distillation in monocular depth estimation (MDE). By utilizing out-of-distribution (OOD) simulated images and a specialized transformation network to bridge the domain gap, the proposed method enables compact student models to learn from complex teacher models without accessing the original, often private, training data.

Introduction

Monocular Depth Estimation (MDE)—the task of predicting 3D depth from a single 2D image—is a cornerstone of robotics and augmented reality. While deep learning has significantly advanced MDE accuracy, state-of-the-art models are frequently too computationally expensive for deployment on edge devices. Knowledge Distillation (KD) typically addresses this by training a compact "student" model to mimic a large "teacher" model.

Overview of the data-free distillation framework using simulated data to train a lightweight model from an expert.

However, standard KD relies on the availability of the original training data, which is often restricted due to privacy, proprietary concerns, or logistical constraints. Existing "data-free" KD methods, which often synthesize images from noise, struggle with the high-resolution, spatially complex nature of depth maps. This paper introduces a framework that leverages simulated data as a proxy to overcome these limitations.

Methodology

The proposed framework uses simulated images to bridge the gap left by missing real-world data. The approach integrates three core components:

  1. Object-wise Mixing: Inspired by ClassMix, this technique randomly swaps objects between simulated images to create diverse training samples, accounting for unknown scene configurations in the original data.
  2. Feature-Based Transformation Network ($G$): To address the domain gap between synthetic and real images, a transformation network adjusts simulated images so their feature statistics match the "running average" statistics stored in the teacher model's Batch Normalization layers.
  3. Joint Optimization: The student model and the transformation network are trained concurrently. The student learns to mimic the teacher's depth predictions, while the transformation network learns to make the synthetic images "look" more like the data the teacher was originally trained on.
Detailed architecture of the proposed joint optimization framework featuring the transformation network and distillation process.

To maintain geometric consistency, an image reconstruction loss ($ℓ_{rec}$) is employed, ensuring the transformation does not distort the underlying structure of the simulated scene.

Experimental Setup

The framework was evaluated against several benchmarks:

  • Target Domains: NYU-v2 and ScanNet (indoor real-world datasets).
  • OOD Source: SceneNet (synthetic indoor trajectories), using subsets of 50,000 and 300,000 images.
  • Baselines: Comparisons were made against OOD data from ImageNet, KITTI, and random Gaussian noise to validate the importance of scene similarity and data scale.

Results and Discussion

The proposed method consistently outperformed baseline data-free approaches and standard KD using raw OOD data.

Key Insights

  • Robustness of Relative Depth: Even when presented with out-of-distribution simulated images, teacher models often produce perceptually accurate relative depth, providing sufficient guidance for student models.
  • Batch Norm as Distribution Memory: Batch Normalization layers serve as a proxy for the original training distribution, allowing the transformation network to "hallucinate" missing domain characteristics.
  • Geometry Preservation: Unlike classification, MDE requires pixel-level structural integrity. The reconstruction loss is critical for successful dense regression in a data-free context.

Quantitative Performance

On the NYU-v2 dataset, the method improved Root Mean Squared Error (RMSE) by approximately 0.05m and increased $\delta_1$ accuracy by about 4% compared to standard KD with OOD data. Furthermore, the framework demonstrated high data efficiency, achieving superior performance with only 1/6th of the training images required by baseline methods.

Conclusion

While the method significantly improves data-free distillation for MDE, a performance gap remains when there is a large capacity difference between teacher and student models. Future work may explore the use of intermediate assistant models or feature-level distillation to further bridge this gap and adapt the framework for diverse outdoor scenarios.

Summary updated

BibTeX
@article{hu2022data,
  author = {Junjie Hu and Chenyou Fan and Mete Ozay and Hualie Jiang and Tin Lun Lam},
  title = {{Data-free Dense Depth Distillation}},
  journal = {Knowledge-Based Systems},
  year = {2022},
}
Figure