Abstract
Diffusion Classifiers (DCs) have emerged as a promising generative alternative to discriminative vision-language models like CLIP, offering better spatial understanding and reduced bias. However, DCs suffer from significant "noise instability," where random noise sampling leads to inconsistent performance and necessitates slow, multi-noise ensembling. We introduce NoOp (Noise Optimization), a method that learns "matching noises" tailored to specific datasets and images. By adhering to principles of Frequency and Spatial Matching, NoOp achieves stable, high-accuracy classification with significantly fewer noise samples, drastically improving inference efficiency.
The Challenge: Noise Instability in Diffusion Classifiers
Traditional discriminative models often fail to capture complex spatial relationships, a limitation known as the "bag-of-words" problem. Diffusion Classifiers address this by framing classification as a conditional denoising task. Given an input image $x_0$ and a sampled noise $\epsilon$, the model predicts the noise $\hat{\epsilon}$ conditioned on various class labels. The label that results in the lowest reconstruction error is chosen as the prediction.
Despite their potential, DCs are plagued by noise instability. Because the initial noise is sampled randomly from a Gaussian distribution, different seeds can produce vastly different classification outcomes. As illustrated in the figure below, different noise samples ($\epsilon_1$ vs $\epsilon_2$) can lead the same classifier to predict entirely different categories for the same image.
To mitigate this, current implementations ensemble results from hundreds of noise samples, which is computationally expensive and slow, often taking nearly 20 seconds per image.
Proposed Solution: NoOp (Noise Optimization)
We propose NoOp, a framework that replaces random noise with optimized "matching" noise. Our approach is built on two key observations:
- Frequency Matching: Different datasets require different spectral focuses. For instance, identifying objects in STL-10 relies on low-frequency shapes, whereas texture classification in DTD requires high-frequency details. NoOp optimizes a dataset-specific noise parameter to capture these essential frequencies.
- Spatial Matching: Informative features are typically concentrated in specific image regions (e.g., the foreground). NoOp employs a lightweight Meta-Network to generate image-specific noise offsets, ensuring the noise targets the most relevant spatial areas.
The final optimized noise is a combination of these dataset-wide and image-specific components. To facilitate training, we use Z-score normalization on classification logits, amplifying the signal-to-noise ratio of the denoising gradients.
Experimental Results
NoOp was evaluated across eight diverse benchmarks, including general object recognition (CIFAR, STL-10), fine-grained classification (Flowers102, OxfordPets), and specialized domains (Medical, Satellite imagery).
- Efficiency and Accuracy: NoOp with just 2-shot training consistently outperformed standard DC ensembling (5x) while being significantly faster.
- Speedup: On ImageNet, NoOp achieved a 3.19% accuracy improvement over ensembling while reducing the total time (training + inference) to approximately 24% of the ensembling baseline.
- Generalization: Noises optimized on ImageNet showed strong transferability, improving performance on most other datasets without further tuning.
Conclusion
Noise is not merely a nuisance in Diffusion Classifiers; it is a critical component that determines classification success. By moving from random sampling to learned, matching noise, NoOp resolves the instability bottleneck of generative classifiers. This work paves the way for more efficient and robust generative perception models that can compete with or exceed the performance of standard discriminative architectures.
Summary updated