Denoising Error as a Proxy for Class Likelihood

33 concepts24 papers1 proposal

Traditional discriminative classifiers learn to draw a boundary between classes. If you train a model to distinguish between a dog and a cat, it looks for the easiest features—the "shortcuts"—to separate the two distributions. If all dogs in the training set happen to be photographed on grass, a standard classifier might simply learn that "grass equals dog."

Generative models work differently. Instead of learning a boundary, they learn the entire data distribution. To generate a realistic image of a dog, a diffusion model must understand the structure of a dog, its texture, its anatomy, and how it relates to its environment.

Recent breakthroughs have shown that we can flip this generative process on its head. Instead of using a text prompt to generate an image, we can use a pre-trained diffusion model to evaluate which text prompt best explains an existing image. This insight turns generative diffusion models into powerful, zero-shot discriminative classifiers.

The Mechanism: Denoising as Likelihood Estimation

How do we extract a classification decision from a model designed to predict noise? The core insight, developed by Alexander Cong Li et al. in Your Diffusion Model is Secretly a Zero-Shot Classifier and concurrently by Kevin Clark et al. in Text-to-Image Diffusion Models are Zero Shot Classifiers, relies on treating the diffusion model's noise-prediction error as a proxy for conditional log-likelihood.

Suppose we have an image $x$ and a set of candidate class labels $Y$ (e.g., "dog", "cat", "car"). The classification process follows a simple, elegant loop for each candidate label $y \in Y$:

  1. Perturbation: We take the input image $x$ and add a known vector of Gaussian noise $\epsilon$ at a specific diffusion timestep $t$, creating a noisy image $x_t$.
  2. Conditional Reconstruction: We feed the noisy image $x_t$ and the timestep $t$ into the diffusion model, conditioned on the candidate class label $y$. The model outputs a prediction of the added noise, $\hat{\epsilon}_\theta(x_t, t, y)$.
  3. Error Evaluation: We compute the reconstruction error—the mean squared error between the true noise $\epsilon$ and the predicted noise $\hat{\epsilon}_\theta$.
                  ┌───────────┐
             ┌───►│ Label: Dog│───┐
             │    └───────────┘   ▼
Input Image ─┤                 Diffusion Model ──► Low Denoising Error (Predicts Dog)
             │    ┌───────────┐   ▲
             └───►│ Label: Cat│───┘
                  └───────────┘

Mathematically, the training objective of a diffusion model is directly related to the variational lower bound of the data log-likelihood. The class-conditioned denoising loss approximates the conditional energy $-\log p(x|y)$.

If the label matches the image (e.g., conditioning on "dog" when the image contains a dog), the model's learned visual prior aligns with the underlying structure beneath the noise, allowing it to predict the perturbation accurately. If the label is incorrect, the text conditioning conflicts with the visual evidence, resulting in a significantly higher prediction error. Assuming a uniform prior over classes, applying Bayes' rule allows us to select the class that minimizes this denoising error.

Why Generative Classifiers Avoid Shortcuts

Standard vision-language models like CLIP rely on contrastive learning, maximizing the embedding similarity between images and text. While highly effective, contrastive models can still fall prey to texture biases or spurious correlations because they only need to learn enough features to distinguish an image from other options in a batch.

Because diffusion classifiers model the full conditional probability of the image given the text, they exhibit fundamentally different properties. As demonstrated by Alexander Cong Li et al. in Generative Classifiers Avoid Shortcut Solutions, forcing a model to account for the generation of the entire image prevents it from relying on simple shortcuts. If a cow appears on a beach instead of a pasture, a discriminative model might misclassify it due to the background shift. A diffusion classifier, however, evaluates how well the word "cow" explains the specific pixels making up the animal, maintaining high robustness to out-of-distribution shifts.

Furthermore, Priyank Jaini et al. in Text-to-Image Diffusion Models are Zero-Shot Classifiers observed that diffusion classifiers exhibit a strong "shape bias" similar to human perception, prioritizing geometric structure over misleading local textures. They also excel at compositional reasoning and attribute binding (e.g., correctly identifying "a blue cube and a red sphere" versus "a red cube and a blue sphere"), a task where contrastive models frequently struggle due to their tendency to treat text prompts as un-ordered bags of words.

Generalizing Across Modalities and Tasks

The paradigm of using generative models for discriminative inference extends far beyond standard 2D natural images:

Nuances and the Domain Gap

While diffusion classifiers offer remarkable zero-shot capabilities and robustness, they introduce a significant computational trade-off. Unlike a standard classifier that requires a single forward pass to output a distribution over thousands of classes, a diffusion classifier requires multiple forward passes per candidate class to average out the stochasticity of the noise injection across different timesteps.

Additionally, their performance depends heavily on how the diffusion process is sampled. In Diffusion Classifiers Understand Compositionality, but Conditions Apply, Arnas Uselis et al. highlighted a critical bottleneck: the "domain gap" between real-world images and the synthetic style distributions the diffusion model was trained to generate. Because text-to-image models are often biased toward clean, aesthetic generations, real-world test images can suffer from an inherent reconstruction penalty.

The authors demonstrated that this gap can be mitigated by intelligently reweighting the diffusion timesteps. By focusing on intermediate timesteps—which capture structural composition and macro-geometry—rather than very low timesteps that demand pixel-perfect texture matching, we can bypass the domain gap and unlock the full potential of generative models for robust visual understanding.

Key Foundational Work

Text-to-Image Diffusion Models are Zero-Shot Classifiers (2023) — contributed 2 concept elements to this direction.
Text-to-Image Diffusion Models are Zero Shot Classifiers (2023) — contributed 2 concept elements to this direction.
Your Diffusion Model is Secretly a Zero-Shot Classifier (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generative_models_for_discriminative_tas_3cb38c2d,
  title = {generative models for discriminative tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-models-for-discriminative-tasks/rdr_3cb38c2d1d344bcf9b6c229e545580d8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_generative_models_for_discriminative_tas_3cb38c2d_v1,
  title = {generative models for discriminative tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-models-for-discriminative-tasks/rdr_3cb38c2d1d344bcf9b6c229e545580d8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake