Standard generative models, particularly diffusion models, are typically trained using supervised objectives like maximum likelihood estimation or score matching. They learn to replicate a target data distribution by reversing a gradual noising process. While this approach is highly effective for capturing the statistical properties of a dataset, it falls short when the goal is to optimize for specific downstream objectives or human preferences. For instance, we might want an image generator to maximize visual aesthetics, a language model to produce logically correct mathematical proofs, or a molecular design model to optimize binding energy.
These downstream objectives are often non-differentiable, black-box functions that can only be evaluated on the final generated sample. If we treat the entire multi-step generation process as a single black-box function, optimizing it via reinforcement learning (RL) leads to extremely high variance and computational inefficiency.
The core insight of this research direction is to frame the iterative generation or denoising process as a multi-step Markov Decision Process (MDP). Instead of treating the model as a single-pass generator, the authors treat it as an RL agent executing a sequence of actions over time. This formulation unlocks the full toolkit of reinforcement learning, enabling precise credit assignment, stepwise optimization, and stable fine-tuning against arbitrary reward functions.
The MDP Formulation of Denoising
To understand why this framework works, we can map the standard components of a reverse diffusion or iterative refinement process directly onto the classic elements of an MDP:
- State ($s_t$): The current noisy or incomplete artifact (e.g., a noisy image, a partially masked sequence of text tokens, or a degraded audio signal) at a specific timestep $t$, along with any conditioning context or prompt.
- Action ($a_t$): The modification applied by the model at the current step (e.g., the predicted noise vector to be subtracted, or the specific tokens to unmask).
- Policy ($\pi_\theta(a_t | s_t)$): The generative model itself, which parameterizes the probability distribution over the next denoising transition.
- Reward ($R$): A feedback signal that evaluates the quality of the generation. This can be a sparse reward evaluated only on the final clean sample $s_0$ (such as human preference or a legal copyright infringement score) or a dense reward distributed across intermediate steps.
By formalizing the generation process this way, Kevin Black et al. introduced Denoising Diffusion Policy Optimization (DDPO) in Training Diffusion Models with Reinforcement Learning, demonstrating that policy gradient methods can directly maximize downstream rewards by leveraging the model's internal probability distribution at every stage of construction. Similarly, Ying Fan et al. developed DPOK in DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models, utilizing online RL with KL regularization to fine-tune text-to-image models based on human feedback without causing the visual over-saturation typical of naive optimization.
This perspective is remarkably general. Early precursors applied sequential decision-making to image enhancement and restoration; for example, Siyu Huang developed an iterative pixel-wise adjustment policy in ReLLIE: Deep Reinforcement Learning for Customized Low-Light Image Enhancement. In modern generative architectures, the framework extends naturally to speech enhancement (Chen Chen et al. in Metric-oriented Speech Enhancement using Diffusion Probabilistic Model), image inverse problems (Haoyue Tang et al. in Solving General Noisy Inverse Problem via Posterior Sampling: A Policy Gradient Viewpoint), and non-autoregressive token generation (Zanlin Ni et al. in AdaNAT: Exploring Adaptive Policy for Token-Based Image Generation).
Overcoming the Computational Bottleneck
A naive implementation of RL over a long denoising trajectory requires backpropagating gradients through the entire sampling chain (Backpropagation Through Time). For a diffusion model with dozens or hundreds of steps, this is computationally prohibitive and highly unstable. The MDP framework provides two primary mechanisms to circumvent this bottleneck:
1. Stepwise Credit Assignment and Denser Rewards
Instead of relying solely on the final output to calculate a reward, the MDP structure allows researchers to utilize intermediate denoising states as training signals. Xiaohang Tang et al. demonstrate in wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models that optimizing at the level of individual steps significantly reduces the number of rollouts required to reach peak performance. To make this process even more efficient, Vishnu Teja Kunde et al. introduce entropy-guided step selection and stepwise advantages in Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages, focusing policy updates exclusively on the most informative denoising steps rather than wasting compute on deterministic stages of the trajectory.
2. Value Functions and Actor-Critic Architectures
By treating the denoising sequence as an MDP, we can train a secondary network—a value function or critic—to estimate the cumulative future reward from any intermediate noisy state. Chen-Xiao Gao et al. show in Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning that a dedicated value function can estimate remaining denoising costs, enabling single-step lookahead optimization. This strategy completely eliminates the need to backpropagate through long diffusion chains. In a similar vein, Lunjun Zhang et al. combine a diffusion-based actor with a distributional critic in D2 Actor Critic: Diffusion Actor Meets Distributional Critic to derive a stable, single-step supervised objective, allowing the model to learn directly from value gradients with minimal computational overhead.
Generalizing to Discrete and Structured Domains
While early successes were concentrated in continuous image generation, the denoising-as-MDP framework has proven highly potent in discrete and highly structured domains like language and structural biology.
In discrete text generation, masked diffusion models generate text by iteratively unmasking or replacing tokens. Chunsan Hong et al. formulate this unmasking process as a KL-regularized MDP in Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies, training a policy that learns optimal generation paths to solve complex reasoning tasks. For broader sequence design, Oussama Zekri et al. introduce Score Entropy Policy Optimization in Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods, which models discrete denoising trajectories as an MDP to facilitate stable fine-tuning against non-differentiable biological or linguistic rewards.
This approach is also revolutionizing molecular inverse design. In RNA 3D structure generation, Tianmeng Hu et al. treat the denoising process of a structural diffusion model as a policy in RIDER: 3D RNA Inverse Design with Reinforcement Learning-Guided Diffusion, optimizing it using reinforcement learning guided by 3D folding rewards to ensure physical and structural accuracy.
Adapting to Few-Step and Distilled Models
As the field shifts toward ultra-fast inference via diffusion distillation and consistency models, the MDP framework has adapted accordingly. When a model generates an image in only one to four steps, the transitions are large and discrete, violating the continuous assumptions of standard diffusion theory.
To address this, Owen Oertell et al. frame the few-step inference of consistency models as a short-horizon MDP in RL for Consistency Models: Faster Reward Guided Text-to-Image Generation, drastically reducing the optimization window and accelerating training speeds up to 17-fold. For distilled models, Risheek Garrepalli et al. treat distillation as an imitation learning problem within an MDP in DDIL: Improved Diffusion Distillation With Imitation Learning, training a student model on its own trajectories to eliminate covariate shift. Furthermore, Zichen Miao et al. mathematically formalize few-step sampling as an MDP to derive pairwise sample optimization losses in Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization, allowing fast models to adapt to human preferences without the massive overhead of full re-distillation.
Key Foundational Work
Distort-and-Recover: Color Enhancement Using Deep Reinforcement Learning (2018) — contributed 1 concept elements to this direction.
Attention-Aware Face Hallucination via Deep Reinforcement Learning (2017) — contributed 1 concept elements to this direction.