Abstract
Adaptation methods have been a workhorse for unlocking the transformative power of pre-trained diffusion models in diverse applications. Existing approaches often abstract adaptation objectives as a reward function and steer diffusion models to generate high-reward samples. However, these approaches can incur high computational overhead due to additional training, or rely on stringent assumptions on the reward such as differentiability. Moreover, despite their empirical success, theoretical justification and guarantees are seldom established. In this paper, we propose DOIT (Doob-Oriented Inference-time Transformation), a training-free and computationally efficient adaptation method that applies to generic, non-differentiable rewards. The key framework underlying our method is a measure transport formulation that seeks to transport the pre-trained generative distribution to a high-reward target distribution. We leverage Doob's $h$-transform to realize this transport, which induces a dynamic correction to the diffusion sampling process and enables efficient simulation-based computation without modifying the pre-trained model. Theoretically, we establish a high probability convergence guarantee to the target high-reward distribution via characterizing the approximation error in the dynamic Doob's correction. Empirically, on D4RL offline RL benchmarks, our method consistently outperforms state-of-the-art baselines while preserving sampling efficiency.
Introduction
Pre-trained diffusion models are incredibly powerful, but out-of-the-box, they primarily generate samples from their original training distribution. Adapting these models for specific downstream tasks—such as maximizing aesthetic quality in images or ensuring task completion in robotics—requires steering them toward a specific "reward."
Existing adaptation methods generally fall into two categories: training-based methods (like RL fine-tuning), which are computationally expensive, and training-free methods, which often require differentiable reward functions. DOIT (Doob-Oriented Inference-time Transformation) addresses these limitations by providing a training-free, computationally efficient algorithm that works with non-differentiable "black-box" rewards while maintaining strong theoretical guarantees.
Methodology
DOIT frames the adaptation problem as transporting the pre-trained generative distribution to a conditional, high-reward target distribution. This is achieved using Doob's $h$-transform, a probabilistic tool that modifies a stochastic process to condition it on a future event.
The DOIT Algorithm
Applying Doob's $h$-transform adds a dynamic correction term ($\nabla \log h$) to the diffusion model's reverse sampling process. DOIT approximates this term through several key mechanisms:
- Monte Carlo Approximation: The correction term is estimated by simulating multiple backward trajectories (rollouts) to identify paths leading to high rewards.
- Bypassing Differentiability: By shifting the gradient operator from the reward function onto the diffusion model's known Gaussian transition density, DOIT can handle rewards from black-box simulators or discrete scorers.
- Surrogate Approximation for Efficiency: To avoid the slowdown of full rollouts at every step, DOIT uses Tweedie's formula to predict the final clean data ($x_0$) from the current noisy state. This allows the reward to be evaluated on predictions instantly, requiring zero additional neural network evaluations compared to standard generation.
Experimental Results
Text-to-Image Generation
Evaluated using Stable Diffusion v1.5 with LAION Aesthetic and ImageReward benchmarks, DOIT's surrogate approximation achieved significant aesthetic improvements. Crucially, it reduced the runtime per image from ~39.5 seconds (for full trajectory simulation) to just ~1.7 seconds, matching the speed of standard sampling.
Offline Reinforcement Learning
On the D4RL locomotion benchmark (HalfCheetah, Hopper, Walker2d), DOIT achieved the highest average score (87.6) among all inference-time methods. It outperformed previous state-of-the-art methods and even surpassed several expensive training-based baselines like Diffuser and QGPO.
Conclusion and Limitations
DOIT offers a robust, mathematically grounded framework for adapting diffusion models at inference time. Its primary limitation is sensitivity to the base model's original distribution; if a high-reward region is extremely rare in the pre-trained model, the Monte Carlo approximation may suffer from high variance. Future research will focus on stabilizing these approximations for extremely out-of-distribution targets.
Summary updated