DreamTeacher: Pretraining Image Backbones with Deep Generative Models

Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, Sanja Fidler

Abstract

DreamTeacher is a novel self-supervised framework that leverages the rich, hierarchical representations learned by pre-trained generative models—such as Diffusion models and GANs—to train standard vision backbones. By distilling generative features into efficient architectures like ResNet and ConvNeXt, DreamTeacher establishes new benchmarks for downstream tasks, particularly in dense prediction scenarios like semantic and instance segmentation. This approach demonstrates that the internal "generative wisdom" required to synthesize realistic images is highly transferable to perception-based computer vision tasks.

Introduction

In modern computer vision, pre-training is a foundational step, typically involving training a model on a massive dataset like ImageNet before fine-tuning it for specific applications. While traditional methods relied on millions of manual labels, Self-Supervised Learning (SSL) emerged to utilize unlabeled data through techniques like Contrastive Learning or Masked Image Modeling (MIM).

However, these methods often overlook a significant resource: Deep Generative Models. To generate a realistic image of a complex scene, a model must internally represent shapes, parts, and spatial hierarchies. DreamTeacher bridges the gap between generative modeling and discriminative perception by figuring out how to transfer this implicit knowledge into the lean, fast backbones used in production environments.

Methodology

The core of the DreamTeacher framework is Knowledge Distillation. Instead of training a backbone from scratch using labels, the "Student" backbone is trained to mimic the internal activations of a pre-trained "Teacher" generative model.

Feature Distillation

The authors utilize a Feature Regressor attached to the student backbone. This regressor learns to map the backbone's features to match the generative model's features at multiple scales. To enhance training efficiency, they employ a combination of Mean Squared Error (MSE) and Attention Transfer, which distills 1D attention maps to guide the student's focus.

Data Regimes

DreamTeacher operates across two primary data regimes depending on the teacher model:

  1. Synthesized Dataset: When using GANs, the framework samples new images directly from the model and records the corresponding features.
  2. Encoded Dataset: When using Diffusion models, real images are processed by adding a controlled amount of noise and "encoding" them back into the model's feature space. This stochastic encoding acts as a powerful, feature-space data augmentation.

Label Distillation

In semi-supervised settings, the framework can be extended by training a small "interpreter" head on the generative model using a limited set of labels. This interpreter then provides "soft labels" to the student, further accelerating the learning process.

Experimental Results

The researchers evaluated DreamTeacher across several diverse benchmarks, including ImageNet-1k, COCO (instance segmentation), ADE20k (semantic segmentation), and BDD100K (autonomous driving).

Performance Benchmarks

DreamTeacher consistently outperformed state-of-the-art (SoTA) SSL methods:

  • Instance Segmentation (COCO): Using a ConvNeXt-B backbone, DreamTeacher achieved 45.2 APmk, outperforming the previous SoTA (iBOT) by +1.0 and the MIM-based SparK by +0.6.
  • Semantic Segmentation (ADE20k): The model reached 42.5 mIoU, significantly beating other self-supervised methods like PixPro (41.6) and SwAV (41.2).
  • Label Efficiency: In low-data regimes, a DreamTeacher-trained ResNet-101 outperformed models with 10x more parameters, demonstrating the potency of generative features when labels are scarce.

Discussion and Insights

Generative Models as Implicit Labelers

Analysis shows that generative models naturally learn semantic clusters. For instance, K-means clustering on StyleGAN2 features reveals that the model distinguishes between categories like "car," "sky," and "tree" without ever receiving explicit supervision. DreamTeacher effectively harvests these latent categories for perception tasks.

Diffusion vs. GANs

While GANs offer speed, Diffusion models (such as ADM) provide richer features for downstream tasks. The stochastic nature of Diffusion encoding provides a robust form of augmentation that makes student backbones more resilient. Furthermore, unlike contrastive learning—which often collapses an image into a single global vector—DreamTeacher preserves spatial information, explaining its superior performance on pixel-level dense tasks.

Limitations and Future Work

Currently, the framework focuses primarily on CNN-based backbones (ResNet, ConvNeXt). Distilling CNN-based generative features into Vision Transformers (ViTs) has proven more challenging and remains a primary target for future research. Additionally, while the distillation process itself is efficient, the initial requirement for a high-quality pre-trained generative model can be computationally expensive for niche datasets. Future iterations may explore cross-resolution distillation to further optimize performance and architectural flexibility.

Summary updated

Abstract

In this work, we introduce a self-supervised feature representation learning framework DreamTeacher that utilizes generative networks for pre-training downstream image backbones. We propose to distill knowledge from a trained generative model into standard image backbones that have been well engineered for specific perception tasks. We investigate two types of knowledge distillation: 1) distilling learned generative features onto target image backbones as an alternative to pretraining these backbones on large labeled datasets such as ImageNet, and 2) distilling labels obtained from generative networks with task heads onto logits of target backbones. We perform extensive analyses on multiple generative models, dense prediction benchmarks, and several pre-training regimes. We empirically find that our DreamTeacher significantly outperforms existing self-supervised representation learning approaches across the board. Unsupervised ImageNet pre-training with DreamTeacher leads to significant improvements over ImageNet classification pre-training on downstream datasets, showcasing generative models, and diffusion generative models specifically, as a promising approach to representation learning on large, diverse datasets without requiring manual annotation.
BibTeX
@inproceedings{li2023dreamteacher,
  author = {Daiqing Li and Huan Ling and Amlan Kar and David Acuna and Seung Wook Kim and Karsten Kreis and Antonio Torralba and Sanja Fidler},
  title = {{DreamTeacher: Pretraining Image Backbones with Deep Generative Models}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure