Noise Matters: Optimizing Matching Noise for Diffusion Classifiers

Yanghao Wang, Long Chen

Abstract

Diffusion Classifiers (DCs) have emerged as a promising generative alternative to discriminative vision-language models like CLIP, offering better spatial understanding and reduced bias. However, DCs suffer from significant "noise instability," where random noise sampling leads to inconsistent performance and necessitates slow, multi-noise ensembling. We introduce NoOp (Noise Optimization), a method that learns "matching noises" tailored to specific datasets and images. By adhering to principles of Frequency and Spatial Matching, NoOp achieves stable, high-accuracy classification with significantly fewer noise samples, drastically improving inference efficiency.

The Challenge: Noise Instability in Diffusion Classifiers

Traditional discriminative models often fail to capture complex spatial relationships, a limitation known as the "bag-of-words" problem. Diffusion Classifiers address this by framing classification as a conditional denoising task. Given an input image $x_0$ and a sampled noise $\epsilon$, the model predicts the noise $\hat{\epsilon}$ conditioned on various class labels. The label that results in the lowest reconstruction error is chosen as the prediction.

The Diffusion Classifier mechanism: adding noise and comparing denoising performance across categories.

Despite their potential, DCs are plagued by noise instability. Because the initial noise is sampled randomly from a Gaussian distribution, different seeds can produce vastly different classification outcomes. As illustrated in the figure below, different noise samples ($\epsilon_1$ vs $\epsilon_2$) can lead the same classifier to predict entirely different categories for the same image.

Noise instability: different random noise samples can lead to incorrect (Samoyed) or correct (Beagle) predictions.

To mitigate this, current implementations ensemble results from hundreds of noise samples, which is computationally expensive and slow, often taking nearly 20 seconds per image.

Proposed Solution: NoOp (Noise Optimization)

We propose NoOp, a framework that replaces random noise with optimized "matching" noise. Our approach is built on two key observations:

  1. Frequency Matching: Different datasets require different spectral focuses. For instance, identifying objects in STL-10 relies on low-frequency shapes, whereas texture classification in DTD requires high-frequency details. NoOp optimizes a dataset-specific noise parameter to capture these essential frequencies.
  2. Spatial Matching: Informative features are typically concentrated in specific image regions (e.g., the foreground). NoOp employs a lightweight Meta-Network to generate image-specific noise offsets, ensuring the noise targets the most relevant spatial areas.

The final optimized noise is a combination of these dataset-wide and image-specific components. To facilitate training, we use Z-score normalization on classification logits, amplifying the signal-to-noise ratio of the denoising gradients.

Experimental Results

NoOp was evaluated across eight diverse benchmarks, including general object recognition (CIFAR, STL-10), fine-grained classification (Flowers102, OxfordPets), and specialized domains (Medical, Satellite imagery).

  • Efficiency and Accuracy: NoOp with just 2-shot training consistently outperformed standard DC ensembling (5x) while being significantly faster.
  • Speedup: On ImageNet, NoOp achieved a 3.19% accuracy improvement over ensembling while reducing the total time (training + inference) to approximately 24% of the ensembling baseline.
  • Generalization: Noises optimized on ImageNet showed strong transferability, improving performance on most other datasets without further tuning.

Conclusion

Noise is not merely a nuisance in Diffusion Classifiers; it is a critical component that determines classification success. By moving from random sampling to learned, matching noise, NoOp resolves the instability bottleneck of generative classifiers. This work paves the way for more efficient and robust generative perception models that can compete with or exceed the performance of standard discriminative architectures.

Summary updated

Abstract

Although today's pretrained discriminative vision-language models (e.g., CLIP) have demonstrated strong perception abilities, such as zero-shot image classification, they also suffer from the bag-of-words problem and spurious bias. To mitigate these problems, some pioneering studies leverage powerful generative models (e.g., pretrained diffusion models) to realize generalizable image classification, dubbed Diffusion Classifier (DC). Specifically, by randomly sampling a Gaussian noise, DC utilizes the differences of denoising effects with different category conditions to classify categories. Unfortunately, an inherent and notorious weakness of existing DCs is noise instability: different random sampled noises lead to significant performance changes. To achieve stable classification performance, existing DCs always ensemble the results of hundreds of sampled noises, which significantly reduces the classification speed. To this end, we firstly explore the role of noise in DC, and conclude that: there are some ``good noises'' that can relieve the instability. Meanwhile, we argue that these good noises should meet two principles: Frequency Matching and Spatial Matching. Regarding both principles, we propose a novel Noise Optimization method to learn matching (i.e., good) noise for DCs: NoOp. For frequency matching, NoOp first optimizes a dataset-specific noise: Given a dataset and a timestep t, optimize one randomly initialized parameterized noise. For Spatial Matching, NoOp trains a Meta-Network that adopts an image as input and outputs image-specific noise offset. The sum of optimized noise and noise offset will be used in DC to replace random noise. Extensive ablations on various datasets demonstrated the effectiveness of NoOp.
BibTeX
@inproceedings{wang2025noise,
  author = {Yanghao Wang and Long Chen},
  title = {{Noise Matters: Optimizing Matching Noise for Diffusion Classifiers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=uG8bjnN1Nl},
}
Figure