Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inference

Denis Blessing, Julius Berner, Lorenz Richter, Carles Domingo-Enrich, Yuanqi Du, Arash Vahdat, Gerhard Neumann

TL;DR

This paper introduces a trust-region framework for stochastic optimal control (SOC) that iteratively transports a path space measure toward a target distribution. By bounding the step-to-step KL divergence, the method automatically constructs a principled geometric annealing schedule with equispaced steps in the Fisher-Rao distance. This approach significantly stabilizes training, controls the variance of importance weights, and prevents mode collapse in high-dimensional sampling and generative model fine-tuning.

What Problem Does This Paper Address?

Stochastic optimal control (SOC) formulations have become a cornerstone of modern machine learning, underpinning everything from diffusion-based sampling to the fine-tuning of generative models. Typically, these tasks are framed as minimizing a divergence (like the reverse KL divergence) between a controlled path measure and a target measure.

However, when the target distribution is complex, multimodal, or far from the initial prior, direct gradient-based optimization fails. In high dimensions, this mismatch leads to extreme variance in importance weights, causing optimization to destabilize or suffer from severe mode collapse (where the model only learns a fraction of the target distribution's modes). While researchers often resort to hand-tuned annealing schedules or expensive preconditioning techniques to bridge this gap, a principled, automated solution has been missing.

Methods

The authors propose Trust Region Constrained Measure Transport. Instead of optimizing for the target measure all at once, the algorithm takes gradual, regulated steps. At each iteration $i$, it solves a constrained optimization problem:

$$u_{i+1} = \arg\min_{u \in \mathcal{U}} D_{\text{KL}}(P^u \parallel Q) \quad \text{s.t.} \quad D_{\text{KL}}(P^u \parallel P^{u_i}) \le \varepsilon$$

where $P^u$ is the path measure of the controlled SDE, $Q$ is the target path measure, and $\varepsilon$ is a strict trust-region bound.

To make this practical, the authors:

  1. Dual Optimization for Multipliers: Convert the constrained problem into an unconstrained saddle-point problem using a relaxed Lagrangian formalism. The optimal Lagrange multiplier $\lambda$ is efficiently computed at each step using a one-dimensional non-linear solver.
  2. Off-Policy Buffer Reuse: Show that the Lagrangian can be framed as a modified SOC problem. This allows the algorithm to reuse trajectories from previous iterations stored in a replay buffer, enabling sample-efficient, off-policy learning.
  3. Practical Instantiations: Apply this framework to two popular SOC objectives, deriving trust-region variants for both the log-variance loss (TR-LV) and the stochastic optimal control matching loss (TR-SOCM).
Figure 1: Illustration of a sequence of distributions (Pui)i resulting from our trust region method (orange) and a mea- sure transport corresponding to non- equispaced geometric annealing (blue), leading to high variance in the impor- tance weights for the initial steps.

Datasets

The method was evaluated across three distinct domains:

  • Diffusion-Based Sampling: Tested on synthetic multimodal targets, including a 10-component Gaussian Mixture Model (GMM), a 40-component GMM (GMM40), and the 32-mode Many Well benchmark.
  • Transition Path Sampling: Evaluated on molecular dynamics tasks using the Alanine Dipeptide isomerization (22 atoms) and Chignolin folding (138 atoms) systems.
  • Fine-Tuning Diffusion Models: Fine-tuned Stable Diffusion 1.5 using human preference prompts from the ImageReward dataset (10,000 training prompts, 100 validation prompts).

Key Insights and Ideas

  • Automated Geometric Annealing: The authors mathematically prove that the trust-region constraint naturally induces a geometric annealing schedule. Crucially, these intermediate steps are automatically equispaced on the statistical manifold according to the Fisher-Rao distance. This eliminates the need for manual hyperparameter tuning of annealing schedules.
  • Controlling Weight Variance: Bounding the step-to-step KL divergence directly controls the variance of the importance weights ($Var(dP^{u_{i+1}}/dP^{u_i}) \approx 2\varepsilon$). Keeping this variance low is what prevents the mathematical explosions typical of high-dimensional importance sampling.

Key Results

Diffusion-Based Sampling

In high-dimensional settings ($d = 200$), standard unconstrained methods completely fail due to mode collapse. The trust-region methods (TR-LV and TR-SOCM) were the only approaches capable of maintaining low control error and accurately estimating the log-normalizing constant across all dimensions.

Figure 2: Performance criteria for a Gaussian mixture target density with varying dimension d, averaged across four seeds. We show the errors of estimating the optimal control, the log-normalizing constant, as well as the Sinkhorn and total variation distances over different dimensions (from left to right). We observe that our trust region methods (TR-SOCM and TR-LV) are the only methods that perform well in high dimensions.

Transition Path Sampling

For molecular dynamics, TR-LV significantly outperformed the baseline Diffusion Path Sampler (TPS-DPS). On the Chignolin folding task, it boosted the Transition Hit Percentage (THP) from 25.00% to 43.95% while dramatically reducing variance across different random seeds, indicating a much more stable training trajectory.

Fine-Tuning Text-to-Image Models

When fine-tuning Stable Diffusion 1.5, TR-SOCM achieved human preference alignment metrics (ImageReward, HPS, CLIP score) comparable to the state-of-the-art Adjoint Matching (AM) baseline. However, by leveraging the replay buffer passes enabled by the trust region, it required three times fewer trajectory simulations, drastically lowering the computational bottleneck of fine-tuning.

Visual comparison of images generated by base vs fine-tuned Stable Diffusion

Limitations and Future Work

The primary limitation of the proposed method is its reliance on a replay buffer that stores entire trajectories. For large-scale applications like text-to-image models, keeping these trajectories in memory introduces a non-trivial computational and data-transfer overhead between the CPU and GPU.

For future work, the authors suggest exploring alternative divergence measures (beyond the KL divergence) for the trust-region constraint and extending this framework to other learned measure transport methods, such as normalizing flows.

Summary updated

Abstract

Solving stochastic optimal control problems with quadratic control costs can be viewed as approximating a target path space measure, e.g. via gradient-based optimization. In practice, however, this optimization is challenging in particular if the target measure differs substantially from the prior. In this work, we therefore approach the problem by iteratively solving constrained problems incorporating trust regions that aim for approaching the target measure gradually in a systematic way. It turns out that this trust region based strategy can be understood as a geometric annealing from the prior to the target measure, where, however, the incorporated trust regions lead to a principled and educated way of choosing the time steps in the annealing path. We demonstrate in multiple optimal control applications that our novel method can improve performance significantly, including tasks in diffusion-based sampling, transition path sampling, and fine-tuning of diffusion models.
BibTeX
@inproceedings{blessing2025trust,
  author = {Denis Blessing and Julius Berner and Lorenz Richter and Carles Domingo-Enrich and Yuanqi Du and Arash Vahdat and Gerhard Neumann},
  title = {{Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inference}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=6RlbOEcOS4},
}
Figure