TL;DR
This paper introduces a trust-region framework for stochastic optimal control (SOC) that iteratively transports a path space measure toward a target distribution. By bounding the step-to-step KL divergence, the method automatically constructs a principled geometric annealing schedule with equispaced steps in the Fisher-Rao distance. This approach significantly stabilizes training, controls the variance of importance weights, and prevents mode collapse in high-dimensional sampling and generative model fine-tuning.
What Problem Does This Paper Address?
Stochastic optimal control (SOC) formulations have become a cornerstone of modern machine learning, underpinning everything from diffusion-based sampling to the fine-tuning of generative models. Typically, these tasks are framed as minimizing a divergence (like the reverse KL divergence) between a controlled path measure and a target measure.
However, when the target distribution is complex, multimodal, or far from the initial prior, direct gradient-based optimization fails. In high dimensions, this mismatch leads to extreme variance in importance weights, causing optimization to destabilize or suffer from severe mode collapse (where the model only learns a fraction of the target distribution's modes). While researchers often resort to hand-tuned annealing schedules or expensive preconditioning techniques to bridge this gap, a principled, automated solution has been missing.
Methods
The authors propose Trust Region Constrained Measure Transport. Instead of optimizing for the target measure all at once, the algorithm takes gradual, regulated steps. At each iteration $i$, it solves a constrained optimization problem:
$$u_{i+1} = \arg\min_{u \in \mathcal{U}} D_{\text{KL}}(P^u \parallel Q) \quad \text{s.t.} \quad D_{\text{KL}}(P^u \parallel P^{u_i}) \le \varepsilon$$
where $P^u$ is the path measure of the controlled SDE, $Q$ is the target path measure, and $\varepsilon$ is a strict trust-region bound.
To make this practical, the authors:
- Dual Optimization for Multipliers: Convert the constrained problem into an unconstrained saddle-point problem using a relaxed Lagrangian formalism. The optimal Lagrange multiplier $\lambda$ is efficiently computed at each step using a one-dimensional non-linear solver.
- Off-Policy Buffer Reuse: Show that the Lagrangian can be framed as a modified SOC problem. This allows the algorithm to reuse trajectories from previous iterations stored in a replay buffer, enabling sample-efficient, off-policy learning.
- Practical Instantiations: Apply this framework to two popular SOC objectives, deriving trust-region variants for both the log-variance loss (TR-LV) and the stochastic optimal control matching loss (TR-SOCM).
Datasets
The method was evaluated across three distinct domains:
- Diffusion-Based Sampling: Tested on synthetic multimodal targets, including a 10-component Gaussian Mixture Model (GMM), a 40-component GMM (GMM40), and the 32-mode Many Well benchmark.
- Transition Path Sampling: Evaluated on molecular dynamics tasks using the Alanine Dipeptide isomerization (22 atoms) and Chignolin folding (138 atoms) systems.
- Fine-Tuning Diffusion Models: Fine-tuned Stable Diffusion 1.5 using human preference prompts from the ImageReward dataset (10,000 training prompts, 100 validation prompts).
Key Insights and Ideas
- Automated Geometric Annealing: The authors mathematically prove that the trust-region constraint naturally induces a geometric annealing schedule. Crucially, these intermediate steps are automatically equispaced on the statistical manifold according to the Fisher-Rao distance. This eliminates the need for manual hyperparameter tuning of annealing schedules.
- Controlling Weight Variance: Bounding the step-to-step KL divergence directly controls the variance of the importance weights ($Var(dP^{u_{i+1}}/dP^{u_i}) \approx 2\varepsilon$). Keeping this variance low is what prevents the mathematical explosions typical of high-dimensional importance sampling.
Key Results
Diffusion-Based Sampling
In high-dimensional settings ($d = 200$), standard unconstrained methods completely fail due to mode collapse. The trust-region methods (TR-LV and TR-SOCM) were the only approaches capable of maintaining low control error and accurately estimating the log-normalizing constant across all dimensions.
Transition Path Sampling
For molecular dynamics, TR-LV significantly outperformed the baseline Diffusion Path Sampler (TPS-DPS). On the Chignolin folding task, it boosted the Transition Hit Percentage (THP) from 25.00% to 43.95% while dramatically reducing variance across different random seeds, indicating a much more stable training trajectory.
Fine-Tuning Text-to-Image Models
When fine-tuning Stable Diffusion 1.5, TR-SOCM achieved human preference alignment metrics (ImageReward, HPS, CLIP score) comparable to the state-of-the-art Adjoint Matching (AM) baseline. However, by leveraging the replay buffer passes enabled by the trust region, it required three times fewer trajectory simulations, drastically lowering the computational bottleneck of fine-tuning.
Limitations and Future Work
The primary limitation of the proposed method is its reliance on a replay buffer that stores entire trajectories. For large-scale applications like text-to-image models, keeping these trajectories in memory introduces a non-trivial computational and data-transfer overhead between the CPU and GPU.
For future work, the authors suggest exploring alternative divergence measures (beyond the KL divergence) for the trust-region constraint and extending this framework to other learned measure transport methods, such as normalizing flows.
Summary updated