A unified perspective on fine-tuning and sampling with diffusion and flow models

Carles Domingo-Enrich, Yuanqi Du, Michael S. Albergo

TL;DR

This paper provides a unified theoretical framework for fine-tuning and sampling from diffusion and flow models using a concept called "exponential tilting." By analyzing various algorithms through a bias-variance decomposition, the authors reveal why certain methods (like Adjoint Matching and Novel Score Matching) enjoy stable, finite gradient variance while other popular methods suffer from catastrophic variance blow-ups at the time boundaries.

What Problem Does This Paper Address?

Generative models like diffusion and flow matching are incredibly successful at learning complex data distributions. However, in many real-world applications, we don't just want to sample from the original data distribution; we want to steer or "tailor" it. This usually happens in one of two ways:

  1. Reward Fine-Tuning: Modifying a pre-trained model (like Stable Diffusion) using a reward function to favor specific qualities (e.g., high aesthetic quality or safety).
  2. Unnormalized Density Sampling: Training a model to sample from a target distribution defined by an energy function where the exact normalizing constant is unknown (common in physics and chemistry).

Historically, these two problems have been studied by different communities using different mathematical tools—namely Stochastic Optimal Control (SOC) and non-equilibrium thermodynamics. This paper bridges the gap, putting these disparate approaches into a single framework so they can be compared on equal footing.

Methods

The authors unify these tasks under the umbrella of exponential tilting, a formulation where a base density ($p^{\text{base}}$) is modified into a target distribution ($p^\star$) via an exponential reward or energy factor: $p^\star(x) \propto p^{\text{base}}(x) \exp(r(x))$.

With this unified view, they analyze and contribute to three main algorithmic paradigms:

  • Stochastic Optimal Control (SOC) Analysis: They provide new mathematical norm bounds on the "lean adjoint ODE" used in Adjoint Matching (AM) and Adjoint Sampling (AS), providing a theoretical foundation for why these methods are so effective.
  • Score Matching Decompositions: They perform a rigorous bias-variance decomposition of several score-matching losses, including Target Score Matching (TSM), Conditional Score Matching (CSM), and Novel Score Matching (NSM).
  • Thermodynamic Formulations: They adapt path-space variational inference methods—specifically Controlled Monte Carlo Diffusions (CMCD) and Non-Equilibrium Transport Samplers (NETS)—to the exponential tilting problem, deriving novel variants of the celebrated Crooks fluctuation theorem and Jarzynski equality.

Datasets

To validate their theoretical insights, the authors conduct reward fine-tuning experiments using:

  • Stable Diffusion 1.5
  • Stable Diffusion 3

They fine-tune these models using 10,000 text prompts from the ImageReward dataset and evaluate performance on a 100-prompt validation slice. The evaluation relies on standard benchmarks including ImageReward, CLIPScore, HPSv2, Aesthetic Score, and DreamSim (to measure image diversity).

Key Insights and Ideas

  • The Boundary Blow-Up (Infinite Variance): The most striking insight comes from the bias-variance decomposition of score-matching methods. The authors prove that under standard importance weights, Target Score Matching (TSM) and Conditional Score Matching (CSM) suffer from infinite gradient variance. This happens because their mathematical formulations force the gradients to blow up at the trajectory boundaries ($t=0$ or $t=1$).
  • Why AM and NSM Win: Conversely, Adjoint Matching/Sampling (AM/AS) and Novel Score Matching (NSM) manage to bypass these boundary blow-ups, maintaining a stable, finite variance throughout training. This explains their strong empirical performance and training stability.
  • Bridging Physics and Machine Learning: By adapting CMCD and NETS, the paper demonstrates that steering a generative model can be mathematically viewed as a thermodynamic process, mapping machine learning loss functions directly to physical concepts like work functionals and free energy differences.

Key Results

The paper's core theoretical finding regarding gradient variance is neatly summarized in their bias-variance decomposition analysis:

Upper bounds on the variance term in the bias-variance decomposition for different flow matching subcases. TSM, CSM, and iDEM suffer from infinite variance (+∞), while AM/AS and NSM remain finite.

In practical text-to-image experiments, the authors successfully fine-tuned Stable Diffusion 1.5 and 3 using Adjoint Matching. They mapped out the Pareto front showing the trade-off between image quality and prompt diversity. Notably, they found that the choice of inference noise ($\eta$) matters: for Stable Diffusion 1.5, turning off inference noise ($\eta=0$) yielded better ImageReward and HPSv2 scores, whereas for Stable Diffusion 3, keeping the memoryless noise schedule ($\eta=1$) proved superior.

Limitations and Future Work

  • Domain Scope: The empirical validation is strictly focused on text-to-image diffusion models. The authors note that testing these frameworks on other vital domains, such as protein design or molecular conformation generation, remains future work.
  • Lack of Empirical Thermodynamic Testing: While the paper introduces beautiful mathematical adaptations of thermodynamics-inspired losses (CMCD and NETS) for exponential tilting, it does not provide experimental evaluations for them, leaving a systematic empirical study for future exploration.
  • Algorithmic Comprehensiveness: The framework, while broad, is not fully exhaustive and does not explicitly incorporate a few other recent online reinforcement learning or sampling formulations.

Summary updated

Abstract

We study the problem of training diffusion and flow generative models to sample from target distributions defined by an exponential tilting of a base density; a formulation that subsumes both sampling from unnormalized densities and reward fine-tuning of pre-trained models. This problem can be approached from a stochastic optimal control (SOC) perspective, using adjoint-based or score matching methods, or from a non-equilibrium thermodynamics perspective. We provide a unified framework encompassing these approaches and make three main contributions: (i) bias-variance decompositions revealing that Adjoint Matching/Sampling and Novel Score Matching have finite gradient variance, while Target and Conditional Score Matching do not; (ii) norm bounds on the lean adjoint ODE that theoretically support the effectiveness of adjoint-based methods; and (iii) adaptations of the CMCD and NETS loss functions, along with novel Crooks and Jarzynski identities, to the exponential tilting setting. We validate our analysis with reward fine-tuning experiments on Stable Diffusion 1.5 and 3.
BibTeX
@misc{domingoenrich2026unified,
  author = {Carles Domingo-Enrich and Yuanqi Du and Michael S. Albergo},
  title = {{A unified perspective on fine-tuning and sampling with diffusion and flow models}},
  year = {2026},
  eprint = {2605.00229},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.00229},
}
Figure