RiDEx: Reward-Informed Directional Exploration for Diffusion Reinforcement Learning

Song Yan, Chenfeng Wang, Wei Zhai, Xingchen Wang, Tao Zhang, Yawen Shao, Shi Pan, Min Li, Zheng-Jun ZhaarXiv 2026 · arXiv

TL;DR

RiDEx (Reward-Informed Directional Exploration) is a novel post-training framework that dramatically accelerates the alignment of diffusion models with specific rewards. By replacing standard random exploration with targeted, reward-relevant directions, RiDEx achieves up to a 29x speedup in training efficiency and sets new state-of-the-art reward peaks across multiple benchmarks.

What Problem Does This Paper Address?

Fine-tuning diffusion models using Reinforcement Learning (RL) is notoriously expensive. To improve a model (e.g., making it better at following prompts like "a cat on a red chair"), RL algorithms must "explore" the latent space to find versions of an image that yield higher rewards.

The core bottleneck is exploration efficiency. Standard methods inject random (isotropic) Gaussian noise to explore. However, in high-dimensional latent spaces, a random direction is almost certainly "orthogonal" to the direction that actually improves the reward. As shown in Figure 1, random perturbations might change the cat's pose or the lighting (yielding zero reward signal) while leaving the chair blue. This "needle in a haystack" problem means most computational budget is wasted on trajectories that reveal nothing about how to improve the task score.

Figure 1. Reward-informed exploration turns targeted changes into learning signals. Left: At equal perturbation norm, unrelated variations leave the chair blue, whereas a reward-relevant change targets the requested color. Center: RiDEx proposes directions, scores paired endpoints, and supervises their originating state. Right: Preference target attainment in H200 GPU hours. The schematic depicts RiDEx-Grad; Eqs. 3, 7, and 11 define the interventions, normalized credit, and CFG update.

Methods

The authors introduce RiDEx, which concentrates exploration into reward-relevant subspaces of the denoising velocity field. Instead of wandering aimlessly, the model "probes" specific directions:

  1. Directional Proposals:
  2. RiDEx-Grad: When a differentiable reward model is available, it uses the reward gradient ($\nabla R$) to propose an exploration direction.
  3. RiDEx-CFG: For non-differentiable tasks (like OCR or counting), it uses the model’s own Classifier-Free Guidance (CFG) residual as a proxy for reward-relevant semantic directions.
  4. State-Matched Credit: From a shared intermediate denoising state, the model generates two "probes" by perturbing in opposite directions along the proposed axis ($+u$ and $-u$). By comparing the terminal rewards of these two resulting images, RiDEx calculates a "signed local supervision" signal.
  5. Hybrid Learning: This local directional signal is combined with global objectives (like DiffusionNFT). This ensures the model learns both from the specific "contrast" of the probes and from the general quality of the generated images.

Datasets

The researchers evaluated RiDEx on Stable Diffusion 3.5 Medium across four diverse alignment and preference tasks:

  • GenEval: Tests compositional reasoning (e.g., object counting and color binding).
  • OCR: Measures the legibility and accuracy of rendered text.
  • HPSv2 (Human Preference Score v2): A differentiable proxy for general human aesthetic and alignment preferences.
  • PickScore: Another differentiable model trained on human over-optimization preferences.

Key Insights and Ideas

  • Dimension vs. Response: Simply reducing the dimensionality of exploration isn't enough; the chosen subspace must capture reward sensitivity. RiDEx proves that exploring along the reward gradient captures significantly more "learning energy" per perturbation unit than random noise.
  • The "Aha" Moment for CFG: The authors found that the CFG residual—the difference between conditional and unconditional denoising—is an excellent derivative-free source of exploration directions for alignment tasks where explicit reward gradients are missing.
  • State Specificity: Learning is most effective when the "credit" (the reward signal) is tied back to the exact state and action that produced it. RiDEx’s symmetric probing acts like a local "compass," telling the model exactly which way to shift its parameters to improve.

Key Results

RiDEx demonstrates massive gains in both efficiency and final performance:

  • Massive Speedups: RiDEx-Grad reached target reward levels 15× faster on HPSv2 and 29× faster on PickScore compared to the previous state-of-the-art, DiffusionNFT, when measured in H200 GPU hours.
  • Higher Peaks: RiDEx achieved the highest peak rewards across all four benchmarks (GenEval, OCR, HPSv2, and PickScore).
  • Qualitative Superiority: As seen in Figure 3, RiDEx successfully fixes common diffusion failures, such as miscounting objects ("five apples"), color bleeding ("red cat and blue dog"), and garbled text in complex scenes.
Figure 2. Reward-efficient learning in H200 GPU hours. Dashed lines mark the shared target in each panel, with attainment measured at the first checkpoint reaching that score. Table 2 gives the corresponding GPU times. RiDEx denotes RiDEx-CFG on GenEval and OCR and RiDEx-Grad on HPSv2 and PickScore.

Limitations and Future Work

While RiDEx significantly improves efficiency, some areas remain for exploration:

  • Reward Model Dependency: The "Grad" version relies on the quality and differentiability of the reward model. If the reward model is biased or easily "gamed," the directional exploration might accelerate that gaming (reward hacking).
  • Computational Overhead: Generating paired probes requires extra denoising steps during training. While the authors show this is more than offset by the massive reduction in total training time, it does increase the cost per training iteration.
  • Future Directions: The authors suggest that extending this directional approach to more complex, multi-objective rewards or integrating it with more advanced policy gradient methods could further push the boundaries of model alignment.

Summary updated

Abstract

Reward-based diffusion post-training updates model parameters from task rewards. While existing approaches improve sampling and policy optimization, they still rely largely on isotropic random perturbations in a high-dimensional latent space for exploration. Such variation often spends the perturbation budget on changes with little influence on the target reward, making exploration inefficient. We study where diffusion models should explore and how to learn from that exploration. We introduce Reward-Informed Directional Exploration (RiDEx), which concentrates local exploration in reward-relevant subspaces of the denoising velocity field. Conditional residuals or differentiable reward gradients propose the subspace, and equal-and-opposite perturbations at the same generation state measure its terminal reward response. The reward difference supplies signed local supervision in the explored coordinates, while all generated images contribute to global DiffusionNFT learning. RiDEx thereby couples exploration-space selection with feedback about the changes it produces. On Stable Diffusion 3.5 Medium, RiDEx attains the highest peak rewards among the evaluated methods on four alignment and preference tasks. Its gradient-direction instantiation achieves 15x and 29x target-attainment speedups over DiffusionNFT on HPSv2 and PickScore, measured in H200 GPU hours. Controlled experiments confirm that reward-aligned directions elicit stronger responses at equal perturbation magnitude, and that applying their measured credit improves curve-average reward across all four tasks.
BibTeX
@misc{yan2026ridex,
  author = {Song Yan and Chenfeng Wang and Wei Zhai and Xingchen Wang and Tao Zhang and Yawen Shao and Shi Pan and Min Li and Zheng-Jun Zha},
  title = {{RiDEx: Reward-Informed Directional Exploration for Diffusion Reinforcement Learning}},
  year = {2026},
  eprint = {2611.22052},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22052},
}
Figure