Diffusion Model as Representation Learner

Xingyi Yang, Xinchao Wang

Abstract

Diffusion Probabilistic Models (DPMs) have recently demonstrated impressive results on various generative tasks. Despite its promises, the learned representations of pre-trained DPMs, however, have not been fully understood. In this paper, we conduct an in-depth investigation of the representation power of DPMs, and propose a novel knowledge transfer method that leverages the knowledge acquired by generative DPMs for recognition tasks. Our study begins by examining the feature space of DPMs, revealing that DPMs are inherently denoising autoencoders that balance the representation learning with regularizing model capacity. To this end, we introduce a novel knowledge transfer paradigm named RepFusion. Our paradigm extracts representations at different time steps from off-the-shelf DPMs and dynamically employs them as supervision for student networks, in which the optimal time is determined through reinforcement learning. We evaluate our approach on several image classification, semantic segmentation, and landmark detection benchmarks, and demonstrate that it outperforms state-of-the-art methods. Our results uncover the potential of DPMs as a powerful tool for representation learning and provide insights into the usefulness of generative models beyond sample generation.

Introduction

While Diffusion Models have revolutionized image generation (e.g., Stable Diffusion, DALL-E), their utility as representation learners—models that "understand" image content for discriminative tasks like classification—has remained largely unexplored. Standard DPMs are difficult to use for recognition due to their time-conditioned U-Net architectures, high computational overhead, and the ambiguity of which diffusion timestep ($t$) provides the most informative features for a specific task.

This research reveals that DPMs are essentially high-powered multi-scale denoising autoencoders. By establishing a theoretical link between DPMs and Denoising Autoencoders (DAEs), the authors introduce RepFusion, a framework that distills meaningful representations from pre-trained diffusion "teachers" into efficient "student" networks.

The RepFusion Framework

To exploit the latent knowledge within DPMs, the researchers propose a knowledge transfer paradigm that addresses the limitations of standard diffusion architectures.

Conceptual overview of RepFusion and the theoretical link between DPMs and Denoising Autoencoders.

RepFusion operates through three key mechanisms:

  1. Knowledge Distillation: A smaller student network (like a ResNet) is trained to mimic features produced by a heavy DPM teacher, specifically pulling from the "MID-BLOCK" of the U-Net where semantic information is most concentrated.
  2. Reinforced Time Selection: Because the optimal timestep $t$ varies per image and task, the framework uses the REINFORCE algorithm. An RL agent learns to select the most informative timestep for each input sample to provide the student with optimal supervision.
  3. Theoretical Equivalence: The paper proves that DPMs are mathematically equivalent to a cascade of DAEs. As noise increases, the model shifts from learning fine-grained details to enforcing "spectral regularization," focusing on global structure.

Understanding the Feature Space

The study provides critical insights into how diffusion models perceive data across different noise levels.

Visualization of attention maps across different diffusion timesteps, showing a shift from local to global dependencies.
  • The "Sweet Spot" of Noise: Visualizations suggest that intermediate steps (e.g., $t=100$ or $t=300$) create cleaner semantic clusters than pure noise ($t=999$) or clean images ($t=0$). A moderate amount of noise helps the model ignore high-frequency details and focus on underlying categories.
  • Attention Shifts: As shown in the attention maps, the model's focus is local at low noise levels. At high noise levels, the model is forced to utilize "long-range dependencies," looking at the entire image to reconstruct structure from chaos.
  • Natural Regularization: The diffusion process acts as a balancer, preventing the model from over-focusing on irrelevant details while capturing meaningful global features.

Experimental Results

RepFusion was evaluated across diverse benchmarks, consistently outperforming standard baselines and self-supervised methods:

  • Semantic Segmentation: On CelebAMask-HQ, a ResNet-18 student reached 67.37 mIOU, a significant improvement over the baseline's 61.22, surpassing established methods like MoCov2 and SwAV.
  • Image Classification: The method provided a +1.83% average accuracy boost on Tiny-ImageNet.
  • Landmark Detection: RepFusion demonstrated superior robustness in "hard cases," such as faces with extreme poses or occlusions, where traditional models often fail.

Conclusion

RepFusion demonstrates that the generative knowledge of Diffusion Models can be effectively repurposed for recognition tasks. While the training phase is computationally intensive due to the heavy teacher model and RL agent, the resulting student networks are efficient and highly capable. This work highlights the potential of generative models to serve as powerful, multi-scale representation learners beyond simple image synthesis.

Summary updated

Abstract

Diffusion Probabilistic Models (DPMs) have recently demonstrated impressive results on various generative tasks. Despite its promises, the learned representations of pre-trained DPMs, however, have not been fully understood. In this paper, we conduct an in-depth investigation of the representation power of DPMs, and propose a novel knowledge transfer method that leverages the knowledge acquired by generative DPMs for recognition tasks. Our study begins by examining the feature space of DPMs, revealing that DPMs are inherently denoising autoencoders that balance the representation learning with regularizing model capacity. To this end, we introduce a novel knowledge transfer paradigm named RepFusion. Our paradigm extracts representations at different time steps from off-the-shelf DPMs and dynamically employs them as supervision for student networks, in which the optimal time is determined through reinforcement learning. We evaluate our approach on several image classification, semantic segmentation, and landmark detection benchmarks, and demonstrate that it outperforms state-of-the-art methods. Our results uncover the potential of DPMs as a powerful tool for representation learning and provide insights into the usefulness of generative models beyond sample generation. The code is available at https://github.com/Adamdad/Repfusion.
BibTeX
@inproceedings{yang2023diffusion,
  author = {Xingyi Yang and Xinchao Wang},
  title = {{Diffusion Model as Representation Learner}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=eaZCYIBAUG},
}
Figure