Training-Free Adaptation of Diffusion Models via Doob's $h$-Transform

Qijie Zhu, Zeqi Ye, Han Liu, Zhaoran Wang, Minshuo Chen

Abstract

Adaptation methods have been a workhorse for unlocking the transformative power of pre-trained diffusion models in diverse applications. Existing approaches often abstract adaptation objectives as a reward function and steer diffusion models to generate high-reward samples. However, these approaches can incur high computational overhead due to additional training, or rely on stringent assumptions on the reward such as differentiability. Moreover, despite their empirical success, theoretical justification and guarantees are seldom established. In this paper, we propose DOIT (Doob-Oriented Inference-time Transformation), a training-free and computationally efficient adaptation method that applies to generic, non-differentiable rewards. The key framework underlying our method is a measure transport formulation that seeks to transport the pre-trained generative distribution to a high-reward target distribution. We leverage Doob's $h$-transform to realize this transport, which induces a dynamic correction to the diffusion sampling process and enables efficient simulation-based computation without modifying the pre-trained model. Theoretically, we establish a high probability convergence guarantee to the target high-reward distribution via characterizing the approximation error in the dynamic Doob's correction. Empirically, on D4RL offline RL benchmarks, our method consistently outperforms state-of-the-art baselines while preserving sampling efficiency.

Introduction

Pre-trained diffusion models are incredibly powerful, but out-of-the-box, they primarily generate samples from their original training distribution. Adapting these models for specific downstream tasks—such as maximizing aesthetic quality in images or ensuring task completion in robotics—requires steering them toward a specific "reward."

Existing adaptation methods generally fall into two categories: training-based methods (like RL fine-tuning), which are computationally expensive, and training-free methods, which often require differentiable reward functions. DOIT (Doob-Oriented Inference-time Transformation) addresses these limitations by providing a training-free, computationally efficient algorithm that works with non-differentiable "black-box" rewards while maintaining strong theoretical guarantees.

Methodology

DOIT frames the adaptation problem as transporting the pre-trained generative distribution to a conditional, high-reward target distribution. This is achieved using Doob's $h$-transform, a probabilistic tool that modifies a stochastic process to condition it on a future event.

The DOIT tilted sampling process uses a dynamic correction term to steer trajectories toward high-reward regions, unlike standard MC simulations.

The DOIT Algorithm

Applying Doob's $h$-transform adds a dynamic correction term ($\nabla \log h$) to the diffusion model's reverse sampling process. DOIT approximates this term through several key mechanisms:

  1. Monte Carlo Approximation: The correction term is estimated by simulating multiple backward trajectories (rollouts) to identify paths leading to high rewards.
  2. Bypassing Differentiability: By shifting the gradient operator from the reward function onto the diffusion model's known Gaussian transition density, DOIT can handle rewards from black-box simulators or discrete scorers.
  3. Surrogate Approximation for Efficiency: To avoid the slowdown of full rollouts at every step, DOIT uses Tweedie's formula to predict the final clean data ($x_0$) from the current noisy state. This allows the reward to be evaluated on predictions instantly, requiring zero additional neural network evaluations compared to standard generation.

Experimental Results

Text-to-Image Generation

Evaluated using Stable Diffusion v1.5 with LAION Aesthetic and ImageReward benchmarks, DOIT's surrogate approximation achieved significant aesthetic improvements. Crucially, it reduced the runtime per image from ~39.5 seconds (for full trajectory simulation) to just ~1.7 seconds, matching the speed of standard sampling.

Offline Reinforcement Learning

On the D4RL locomotion benchmark (HalfCheetah, Hopper, Walker2d), DOIT achieved the highest average score (87.6) among all inference-time methods. It outperformed previous state-of-the-art methods and even surpassed several expensive training-based baselines like Diffuser and QGPO.

Conclusion and Limitations

DOIT offers a robust, mathematically grounded framework for adapting diffusion models at inference time. Its primary limitation is sensitivity to the base model's original distribution; if a high-reward region is extremely rare in the pre-trained model, the Monte Carlo approximation may suffer from high variance. Future research will focus on stabilizing these approximations for extremely out-of-distribution targets.

Summary updated

Abstract

Adaptation methods have been a workhorse for unlocking the transformative power of pre-trained diffusion models in diverse applications. Existing approaches often abstract adaptation objectives as a reward function and steer diffusion models to generate high-reward samples. However, these approaches can incur high computational overhead due to additional training, or rely on stringent assumptions on the reward such as differentiability. Moreover, despite their empirical success, theoretical justification and guarantees are seldom established. In this paper, we propose DOIT (Doob-Oriented Inference-time Transformation), a training-free and computationally efficient adaptation method that applies to generic, non-differentiable rewards. The key framework underlying our method is a measure transport formulation that seeks to transport the pre-trained generative distribution to a high-reward target distribution. We leverage Doob's $h$-transform to realize this transport, which induces a dynamic correction to the diffusion sampling process and enables efficient simulation-based computation without modifying the pre-trained model. Theoretically, we establish a high probability convergence guarantee to the target high-reward distribution via characterizing the approximation error in the dynamic Doob's correction. Empirically, on D4RL offline RL benchmarks, our method consistently outperforms state-of-the-art baselines while preserving sampling efficiency.
BibTeX
@inproceedings{zhu2026training,
  author = {Qijie Zhu and Zeqi Ye and Han Liu and Zhaoran Wang and Minshuo Chen},
  title = {{Training-Free Adaptation of Diffusion Models via Doob's $h$-Transform}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=5cu1ynhwTJ},
}
Figure