Repurposing Pretrained Diffusion Models for Visual Perception

282 concepts189 papers2 proposals

For years, the computer vision community relied heavily on discriminative pretraining—such as ImageNet classification or CLIP's contrastive learning—to build visual feature extractors. While these models excel at capturing global semantic concepts, they often discard fine-grained spatial details because their training objectives force them to compress an entire image into a single abstract vector.

Conversely, denoising diffusion probabilistic models are trained to do the exact opposite: reconstruct images pixel by pixel from pure noise. To successfully remove noise at multiple scales, a diffusion model's internal architecture—typically a U-Net—must simultaneously track high-level semantic context (to know what it is generating) and precise low-level geometry (to know where every edge, texture, and boundary belongs). This insight has led to a powerful realization: frozen, pretrained diffusion models are secret, highly effective dense representation learners.

Mechanics of Feature Extraction

How do we extract representations from a model designed for generation? Instead of running the full, computationally expensive iterative reverse denoising loop, we can treat the diffusion U-Net as a single-pass feature extractor.

In the foundational work by Dmitry Baranchuk et al. in Label-Efficient Semantic Segmentation with Diffusion Models, the authors demonstrated that injecting a controlled amount of noise into an image and performing a forward pass through the U-Net exposes highly descriptive intermediate activations. Specifically, the decoder blocks of the U-Net yield high-dimensional vectors that map perfectly to local pixel information, serving as dense descriptors for discriminative tasks.

Input Image ---> [ Add Noise (t) ] ---> [ Diffusion U-Net ] ---> Extract Intermediate Activations ---> Downstream Task Head

To avoid the stochasticity and potential distortion of adding noise entirely, Wenliang Zhao et al. introduced the VPD framework in Unleashing Text-to-Image Diffusion Models for Visual Perception. They showed that a single forward pass at timestep $t=0$ without any added noise can extract rich visual-linguistic representations directly from a text-to-image diffusion model's U-Net. This clean, single-pass feature extraction allows the network to inherit generative visual priors while serving as an immediate backbone for downstream perception tasks like depth estimation and semantic segmentation.

The Two Axes of Representation: Layers and Timesteps

Unlike standard convolutional networks or vision transformers that vary features only across network depth, diffusion models possess a unique second axis of variation: the denoising timestep $t$. This creates a multi-scale representation space where different combinations of layers and timesteps capture distinct visual properties.

  • The Layer Axis: Deep bottleneck layers capture abstract semantic categories and global context, while shallow encoder and decoder layers preserve fine-grained textures, local geometry, and exact pixel boundaries.
  • The Timestep Axis: Early timesteps (high noise) force the network to focus on global layout and coarse structures. Late timesteps (low noise or $t=0$) allow the network to resolve clean, high-fidelity spatial details and surface textures.

As explored by Ziyi Li et al. in Open-vocabulary Object Segmentation with Diffusion Models, extracting features at a late denoising stage strikes an optimal balance between semantic clarity and spatial precision, allowing secondary modules to leverage the latent spatial knowledge of the frozen generative model. Similarly, Bahar Aydemir in Data Augmentation via Latent Diffusion for Saliency Prediction notes that the U-Net's intermediate features naturally decouple into low-level readouts for color and contrast and high-level readouts for object semantics, making the frozen generative model a comprehensive feature extractor without any backbone retraining.

Robustness to Domain Shifts and Modalities

Because diffusion models are trained on massive, diverse datasets to model the entire pixel distribution, their internal features exhibit remarkable robustness against domain shifts, abstract styles, and cross-modal variations that typically break standard vision models.

When dealing with non-photorealistic or highly stylized inputs, standard vision backbones often fail due to a strong bias toward realistic textures. However, Li Wang et al. demonstrated in Sketch2PoseNet: Efficient and Generalized Sketch to 3D Human Pose Prediction that diffusion priors capture high-level structural information far more effectively from abstract visual data like sketches. This structural resilience is echoed by Siddharth Tourani in Pose-Guided Self-Training with Two-Stage Clustering for Unsupervised Landmark Discovery, where frozen diffusion features implicitly retain geometric and part-level information, enabling unsupervised landmark discovery without any manual annotations.

Furthermore, these features provide a powerful mechanism for establishing semantic correspondence across entirely different images or modalities. Jisu Nam et al. leveraged this in DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization, using internal feature maps as dense descriptors to align object parts across complex, non-rigid transformations. In cross-modal scenarios, Yepeng Liu et al. showed in MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching that intermediate layers focus heavily on object identity rather than sensor-specific noise or illumination, facilitating zero-shot multimodal matching between disparate data types like SAR and optical imagery.

Unlocking Label Efficiency

The practical consequence of these rich, pre-trained generative representations is extreme label efficiency. Because the frozen diffusion backbone already understands scene structure and semantics, downstream tasks can be solved using lightweight "linear probes" or shallow adapter modules trained on a fraction of the usual data.

In the medical domain, where labeled data is notoriously scarce, Héctor Carrión et al. developed the FEDD framework in FEDD - Fair, Efficient, and Diverse Diffusion-based Lesion Segmentation and Malignancy Classification, showing that linear probes on top of frozen diffusion features achieve state-of-the-art lesion segmentation using only 5-10% of standard training labels. Similarly, Chun-Mei Feng et al. demonstrated in Enhancing Label-efficient Medical Image Segmentation with Text-guided Diffusion Models that extracting intermediate activations during the reverse process allows for high-quality segmentation with minimal labeled samples by leveraging cross-modal text alignments.

Beyond segmentation, this paradigm extends to complex counting and localization tasks. Xiaofei Hui et al. showed in Class-Agnostic Object Counting with Text-to-Image Diffusion Model that the internal attention maps of text-to-image models contain sufficient instance-level awareness to count novel objects in a zero-shot or few-shot manner. This proves that the features required to synthesize an image are inherently robust enough to parse, locate, and count its constituents.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 191 papers
  2. Version 1 · 2026-10-04 · 240 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_diffusion_based_feature_extraction_8bd1ca5f,
  title = {diffusion-based feature extraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/diffusion-based-feature-extraction/rdr_8bd1ca5f59a0469ca23614aa38bf6a94},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_diffusion_based_feature_extraction_8bd1ca5f_v2,
  title = {diffusion-based feature extraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/diffusion-based-feature-extraction/rdr_8bd1ca5f59a0469ca23614aa38bf6a94/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake