Self-Improving Diffusion Classifiers with Minority Preference Optimization

Hyunsoo Kim, Jungmyung Wi, Soobin Um, Donghyun Kim, Suhyun KimarXiv 2026 · arXiv

TL;DR

Diffusion classifiers excel at zero-shot classification but suffer from a severe bias toward the "majority" data distributions they saw during pretraining. This paper introduces MiPO (Minority Preference Optimization), a self-improving framework that fine-tunes a diffusion model using online reinforcement learning and a novel minority-seeking reward function. Crucially, MiPO requires only text prompts—no additional images or external reward models—to expand the model's perceptual coverage and significantly boost zero-shot classification accuracy on underrepresented data.

What Problem Does This Paper Address?

"Diffusion Classifiers" are a clever twist on generative models: instead of generating images, they use a pretrained text-to-image model's noise-prediction error as a proxy to perform zero-shot classification. While robust against out-of-distribution data, they inherit a fundamental flaw from their generative roots: they are heavily biased toward high-density, "majority" regions of the data manifold.

For instance, if a standard diffusion model is excellent at generating typical "soap dispensers," a diffusion classifier will easily recognize them. However, it will likely misclassify underrepresented, low-density variations (like a soap dispenser shaped like a duck or a lotion bottle) because the underlying model lacks a strong internal representation of these "minority" concepts.

Fig. 1: MiPO expands the perceptual coverage of diffusion classifiers. A stan- dard diffusion classifier reliably recognizes majority (i.e., high-density) samples that are well covered by the pretrained generative distribution (e.g., soap dispenser), but often fails on underrepresented (i.e., low-density) concepts such as lotion, duck, and sink. This illustrates that the recognition ability of diffusion classifiers is closely tied to the generative coverage of the underlying diffusion model. By optimizing minority pref- erence rewards, MiPO encourages the model to better cover such low-density regions, thereby improving the classifier’s perception of minority visual categories without re- quiring additional downstream image data.

Prior works attempted to fix this by steering generation toward minority samples at inference time, but these methods are computationally expensive and don't update the model's internal representations. This paper answers a key question: Can we fundamentally improve a diffusion classifier's perception by training it to better explore minority data regions?

Methods

The authors propose Self-Improving Diffusion Classifiers with Minority Preference Optimization (MiPO). The core idea is to let the model self-train using text prompts to find and reinforce its own weak spots.

The pipeline consists of four main components:

  1. Minority Preference Reward: To quantify how "underrepresented" a sample is without an external reward model, MiPO adds noise to a generated image $x_0$ at an intermediate step ($t=0.9T$) and tries to reconstruct it ($\hat{x}_0$) using a DDIM-based approximation. Images with higher reconstruction errors ($\|x_0 - \hat{x}_0\|_2$) signify regions where the denoiser is highly uncertain—meaning it's a minority concept.
  2. Group Relative Policy Optimization (GRPO): Instead of using a heavy critic model, MiPO samples a group of trajectories from the same prompt and initial noise. It normalizes the rewards within this group to compute advantages, making the reinforcement learning process highly stable and invariant to reward scaling.
  3. LoRA Fine-Tuning: To keep training efficient and modular, updates are applied via Low-Rank Adaptation (LoRA) adapters rather than modifying the full backbone.
  4. Selective Policy Optimization & KL Regularization: To prevent the model from completely forgetting how to generate majority samples (or degrading image quality), the authors apply a Kullback–Leibler (KL) penalty. Furthermore, they only update the policy during the initial 40% of the denoising timesteps, where coarse semantic and structural layouts are formed.
Fig. 2: Training pipeline of Minority Preference Optimization (MiPO). MiPO is integrated into a pretrained text-to-image diffusion model, where multiple SDE tra- jectories are sampled using the same prompt and initial noise. Only the early denoising steps are updated through Selective Policy Selection. For each generated sample, the Minority Preference Reward is obtained from DDIM-based minority scores, which re- flects how strongly the sample lies in a minority region. Rewards within each prompt group are aggregated to compute group-normalized advantages via GRPO, and the final objective combines the minority reward with KL regularization to maintain con- sistency with the original model. MiPO guides the model toward better coverage of minority regions while preserving a stable denoising.

Datasets

  • Training: MiPO uses over 100k text prompts from the HPSv2 dataset (curated from DiffusionDB and MSCOCO). No images are used during training.
  • Evaluation:
  • Zero-shot Classification: CIFAR10, Tiny-ImageNet, and robustness benchmarks including CIFAR10-C, Caltech, and SUN09.
  • Minority Image Generation: Evaluated using 1,000 random captions from the MS-COCO validation set.

Key Insights and Ideas

  • "What I cannot create, I do not understand": Borrowing from Richard Feynman, the authors demonstrate a direct, causal link between a diffusion model's generative coverage and its classification perception. Fixing generation directly fixes classification.
  • Image-Free Self-Improvement: MiPO doesn't need a dataset of target images or a massive foundation model (like CLIP) to act as a reward function. The model uses its own internal denoising uncertainty as the reward signal.
  • Prompt-Adaptive Generation: Unlike previous state-of-the-art methods (like MinorityPrompt) that require slow, per-prompt optimization at inference time, MiPO internalizes minority concepts into a LoRA adapter. Once trained, it generates minority-rich images instantly.

Key Results

  • Classification Boosts: MiPO consistently improved zero-shot classification across multiple backbones. On Stable Diffusion 1.5, it achieved accuracy gains of up to 3.8% on standard benchmarks and 2.0% on corrupted datasets (CIFAR10-C).
  • Targeted Minority Improvement: When isolating the most difficult "minority" test images in CIFAR10, MiPO boosted accuracy from 73.07% to 81.68% (an 8.61% absolute jump).
  • Inference Speedup: For minority image generation, MiPO runs at the same speed as standard DDIM (1 second per image), whereas the previous best method, MinorityPrompt, takes 63 seconds per image due to test-time optimization.
  • Aesthetic Preservation: Human preference evaluations (via ImageReward) showed that MiPO actually improved overall image quality scores compared to the baseline, avoiding the visual degradation typical of aggressive minority-sampling techniques.

Limitations and Future Work

While MiPO excels on clean benchmarks, it struggles on complex, cluttered, or inconsistently annotated datasets like LabelME and VOC2007, where multi-object scenes introduce too much uncertainty into zero-shot evaluation.

Additionally, there is an inherent trade-off between pushing a model to explore extreme minority concepts and maintaining the high-level aesthetic priors of the original pretrained model. Striking the perfect balance remains an open challenge for future research.

Summary updated

Abstract

Prior studies have demonstrated that diffusion classifiers achieve robust zero-shot classification performance. However, their effectiveness is strongly tied to the pretraining data distribution: they perform well in majority, high-density regions of the data manifold, but are significantly less accurate in minority, low-density regions. Although prior works on minority sampling have focused on generating more minority-like images, what minority sampling fundamentally enables beyond generation remains underexplored. In this paper, we reveal a direct relationship between minority sampling in generation and the perception capability of diffusion classifiers. Specifically, we show that enhancing minority sampling broadens the coverage of underrepresented regions on the data manifold, thereby improving diffusion-based recognition. To exploit this connection, we propose \textit{Self-Improving Diffusion Classifiers with Minority Preference Optimization} (MiPO), which fine-tunes a pretrained diffusion model using minority preference rewards. Using only arbitrary caption data, MiPO generates candidate samples, rewards those that better cover minority regions, and optimizes the model with LoRA and Group Relative Policy Optimization, without additional image data, external foundation models, or external reward models. This enables stable, prompt-adaptive minority sampling and translates low-density generative coverage into improved zero-shot diffusion classification. To sum up, we show that diffusion classifier perception is biased toward majority regions, demonstrate that this bias can be alleviated through minority preference optimization, and evaluate MiPO on five standard datasets.
BibTeX
@misc{kim2026self,
  author = {Hyunsoo Kim and Jungmyung Wi and Soobin Um and Donghyun Kim and Suhyun Kim},
  title = {{Self-Improving Diffusion Classifiers with Minority Preference Optimization}},
  year = {2026},
  eprint = {2607.03770},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.03770},
}
Figure