Many recent samplers treat drawing samples from a hard distribution as a steering problem. A neural network supplies a push, called the control, to a noisy particle so that by the end of a fixed time window the particle lands in the target distribution. The network is trained against targets computed from simulated trajectories. In most methods those targets are importance-weighted averages: each simulated path gets a weight proportional to the exponential of its negative cost. This is where the curse of dimensionality comes back. Grid-free methods avoid the exponential cost of discretizing space. But Error Analysis of Sampling Algorithms for Approximating Stochastic Optimal Control proves that importance-sampling estimates of the optimal control have mean squared error growing like $O(d(\sqrt{1.33})^d)$ in state dimension $d$. An interacting particle system in the style of the ensemble Kalman filter reaches $O(d/N)$ with $N$ particles. That proof covers only a single-stage, linear-quadratic problem, and no neural sampler uses the weight-free estimator.
We propose to build and test exactly that combination. The importance-weighted control targets inside a short-horizon neural control learner are replaced with targets from an ensemble-Kalman-style interacting particle update. Instead of reweighting independent paths, a small cloud of particles is moved toward low cost using their empirical covariance with the cost. We also add a correction step that restores the exact target when the Gaussian assumption behind Kalman updates fails. The central question is empirical and falsifiable. On multi-step and nonlinear sampling benchmarks, does the linear-in-$d$ error scaling survive? Does the bias of the Kalman update on multimodal targets cost more than the variance it removes?
Core Claim
In path integral control, the optimal control at $(t,x)$ can be written as a weighted average over simulated noise increments:
$$u^*(t,x)\,dt \approx \frac{\mathbb{E}\left[e^{-S(\tau)/\lambda}\, dw_t\right]}{\mathbb{E}\left[e^{-S(\tau)/\lambda}\right]}.$$
Here $S(\tau)$ is the accumulated cost of a simulated path $\tau$ started at $x$, $\lambda$ is the noise-temperature constant from the Hopf–Cole transform, and $dw_t$ is the first Brownian increment. Every method in the file that estimates this ratio by averaging exponentiated weights inherits the weight degeneracy, whether the average is taken over full trajectories or short branches. The interacting particle alternative writes the same quantity as the endpoint of a tempering flow. The weight exponent goes from $0$ to $1/\lambda$ in small increments, and at each increment the particles' noise increments are moved by a Kalman-type gain: the empirical covariance between increments and cost, times the step size. For linear dynamics and quadratic cost this reproduces the optimal control exactly with error $O(d/N)$. Our claim is that using these weight-free estimates as regression targets for a policy network $u_\phi(t,x)$, over short branches, keeps near-linear scaling with $d$ well past the 80–200 dimensions where current methods fail. We further claim that a light importance correction on top of the transported particles recovers accuracy on non-Gaussian targets, at a small fraction of the variance of plain reweighting.
Relation to Prior Work
Three lines of work motivate the proposal by showing the weight problem in practice. Path Integral Sampler trains a drift network on full rollouts and uses trajectory importance weights for unbiased normalizing-constant estimates. PIPS applies path-integral cross-entropy reweighting to molecular transition paths and reports high Monte Carlo variance on barrier crossings. Stochastic Optimal Control Matching (SOCM) turns control learning into least-squares regression and adds learned matrices $M_t$ that reduce gradient variance. That variance reduction does not touch its importance factor $\alpha$, which explodes for large costs, and SOCM ran out of memory at $d=80$. All three keep exponential weights; we remove them from the target.
The closest method, which we reuse, is Path Integral Value Matching (PI-VM). It breaks the full-trajectory bottleneck with a recursive, temporal-difference form of the path integral over roughly 8-step branches. It scales to 200 dimensions on quadratic Ornstein-Uhlenbeck (OU) problems and recovers a 50-dimensional Many-Well landscape. PI-VM shortens the horizon over which weights accumulate, but it still reweights within each branch. The single-stage analysis above says a one-step exponential weight already degrades exponentially in $d$. We keep PI-VM's short branches and replay buffer and change only the estimator inside a branch. We regress a control directly rather than a value, which also sidesteps PI-VM's inference-time autodifferentiation cost as a side effect, though that is not our main aim.
The theoretical source, which constrains the design, is Error Analysis of Sampling Algorithms. Its guarantee for interacting particles is single-stage and linear-quadratic. Kalman updates are exact only for Gaussian structure, so on multimodal targets the method could collapse modes. For that reason the correction step and multimodal benchmarks are central to the proposal, not optional.
On-policy Girsanov gradients (An Efficient On-Policy Deep Learning Framework) and continuous-time actor-critic flows (Actor-Critic Flow) avoid differentiating through the solver. They address a different bottleneck, memory and backpropagation, and serve here as reference points rather than competitors on the weight question.
Research Program
The first step is a multi-step, nonlinear extension of the interacting particle estimator. From a replay-buffer state $(t,x)$ we launch $N$ branches of length $K$ under the current policy. We then run a tempering schedule on the branch noise. At each stage, each particle's increments are nudged by the empirical increment–cost covariance, and the branch is resimulated with the new increments; nonlinearity enters through this resimulation. The branch terminal is bootstrapped with a target network's value estimate, as in PI-VM. The mean of the transported first increments, divided by $dt$, becomes the regression target for $u_\phi(t,x)$.
The second step is the correction. After transport, we compute importance weights of the target path measure relative to the transported particle distribution. These should be close to uniform when the Gaussian approximation is good. We use them either directly (a self-normalized corrected target) or as a switch that falls back to plain reweighting when effective sample size is high.
The third step is analysis. We extend the $O(d/N)$ bound from one stage to $K$ steps for linear-quadratic dynamics and characterize how bias scales with a measure of non-Gaussianity of the branch cost.
Baselines
The naive reference is the same short-branch learner with plain self-normalized importance weights, which is the MPPI-style estimator. It is run with identical branch length, particle count, network, replay buffer, and wall-clock budget. Its error curve against $d$ is the thing we must bend. Beating it means a clearly shallower slope in the log-error versus $d$ plot on quadratic OU, not just a lower error at one $d$.
The strongest existing method is PI-VM, reimplemented or run from released settings. It has the same particle budget per update, the same branch length (8 steps as reported, plus a sweep), and the same number of gradient steps. It gets an equal hyperparameter search of 20 trials per method per dimension. The file gives only qualitative results: PI-VM scaled to 200 dimensions where SOCM hit memory limits at 80 and Adjoint Matching diverged at 200, and it was 10–20x faster than those methods. No reported numeric L2 control error for PI-VM at $d=200$ was found on Lacuna, so we will measure it ourselves. Beating PI-VM means lower $L^2$ control error against the analytic optimum at matched compute for $d \ge 200$, and continued convergence at $d=400$ and $800$.
SOCM is included as the strongest policy-matching method. It reportedly achieved roughly 10x lower $L^2$ error than adjoint and cross-entropy losses on OU problems, and we expect it to hit memory limits early. For low-dimensional calibration, the actor-critic flow reports 4.3–4.8% relative control error on a 10D linear-quadratic problem (Actor-Critic Flow), and we expect to match that regime.
Two ablations isolate the key ingredient. The first uses Kalman transport without correction, which measures bias. The second uses correction weights applied to untransported particles, which is plain reweighting with our effective-sample-size switch.
Why not just…?
Why not just add resampling, as in sequential Monte Carlo? Resampling redistributes particles after weights degenerate but does not stop the degeneracy within a single step. The single-stage bound in Error Analysis of Sampling Algorithms applies before any resampling could act. This alternative is a real competitor at moderate $d$, so we include a sequential Monte Carlo variant of the naive baseline (tempering plus resampling, matched particles) and compare effective sample size and target error across $d$.
Why not just shorten PI-VM's branches further? Shorter branches reduce accumulated weight variance but increase reliance on the bootstrapped value, trading variance for bias. Per-step weights still degrade with $d$. Our branch-length sweep ($K \in \{1,2,4,8,16\}$) at each $d$ tests this directly. If PI-VM with $K=1$ matches our method at $d=400$, the idea is unnecessary and we will report that.
Why not just use SOCM's learned reparameterization matrices to cut variance? The $M_t$ matrices provably reduce the variance term without changing the optimum, but the importance factor $\alpha$ lies outside their reach. SOCM also ran out of memory at $d=80$ in the comparison reported by PI-VM. SOCM appears in our dimension sweep as long as it fits in memory.
Why not just build the score of the target into the network, as PIS-Grad does? That improves the policy's starting point, which shrinks weight variance, and it combines with our estimator rather than competing with it. We run both estimators with and without gradient-informed parameterization, so a gain that appears only without it would be exposed.
Proposed Experiments
The decisive check is on quadratic and linear OU problems with analytic optimal controls. We sweep $d \in \{10, 40, 80, 200, 400, 800\}$ at fixed $N$ and report control $L^2$ error, target-estimator mean squared error measured separately from training, memory, and wall-clock time. The prediction, derived from the $O(d/N)$ interacting-particle rate, is roughly linear growth for our targets and upward curvature for reweighted targets. Upward curvature for our targets would falsify the claim.
The second check probes where Gaussian updates should break: the 50-dimensional Many-Well task used by PI-VM and the 1024-mode Double Well used by SOCM. Here we measure mode coverage (the fraction of modes populated relative to the ground-truth weights) and control $L^2$ error. The uncorrected ablation is expected to lose modes. The corrected method must match PI-VM's coverage to justify itself.
Third, we test transfer to realistic sampling on the 10D Funnel and the 1,600-dimensional Log Gaussian Cox Process from Path Integral Sampler. We report bias and variance of the log-normalizing-constant estimate against PIS-Grad and PI-VM. For the Funnel we also report Maximum Mean Discrepancy, where the on-policy framework reports results comparable to state-of-the-art baselines but no numeric value was found on Lacuna.
The project succeeds if the corrected estimator beats PI-VM at matched compute for $d \ge 400$ on OU while matching its Many-Well coverage. It fails if the bias of the Kalman transport or the cost of the correction erases the variance gain.
References
- Error Analysis of Sampling Algorithms for Approximating Stochastic Optimal Control
- Path Integral Sampler: A Stochastic Control Approach For Sampling
- PIPS: Path Integral Stochastic Optimal Control for Path Sampling in Molecular Dynamics
- Stochastic Optimal Control Matching
- Path Integral Value Matching for Linear Quadratic Stochastic Optimal Control
- An Efficient On-Policy Deep Learning Framework for Stochastic Optimal Control
- Solving Time-Continuous Stochastic Optimal Control Problems: Algorithm Design and Convergence Analysis of Actor-Critic Flow