In-Context Policy Evaluation for High-Dimensional Nonlinear Control

Policy iteration solves an optimal control problem by alternating two steps. First it scores the current controller: it computes the long-run cost of following that controller from every state. Then it improves the controller by acting greedily with respect to that score. The scoring step is the expensive one. Every neural method that runs policy iteration on high-dimensional nonlinear systems today trains a fresh network at every iteration, for every new plant. We propose a pretrained network that performs the scoring step in context. It reads a batch of recorded trajectories from an unseen nonlinear system running the current controller, and in one forward pass it returns the value function and the improved controller. Policy iteration then becomes a short loop: roll out, read, improve, repeat. No weights are updated at test time, and nothing is ever represented on a grid.

This matters for two reasons. First, the curse of dimensionality in optimal control (the $M^d$ blow-up of grid-based dynamic programming) is usually addressed per instance, by mesh-free networks that each cost minutes to hours to train. Second, in-context operator learning, meaning models that adapt to a new equation from example input/output pairs placed in their input with no retraining, has so far been shown only on 1D and 2D differential equations. Joining the two would let one pretrained model solve many 10–30 dimensional nonlinear control problems in seconds. The policy iteration loop also gives a built-in way to catch the model's own errors.

Core Claim

The piece to amortize is the linear policy-evaluation equation, not the full nonlinear Hamilton-Jacobi-Bellman (HJB) equation or the dynamics. We consider control-affine plants $\dot x = f(x) + G(x)u$ with state $x \in \mathbb{R}^d$ and running cost $\ell(x) + u^\top R u$. For a fixed controller $\pi_k$, the value function $V_k$ of that controller solves the generalized HJB equation

$$\nabla V_k(x)^\top \big(f(x) + G(x)\pi_k(x)\big) + \ell(x) + \pi_k(x)^\top R\,\pi_k(x) = 0 ,$$

which is linear in $V_k$. Improvement is then closed-form:

$$\pi_{k+1}(x) = -\tfrac12 R^{-1} G(x)^\top \nabla V_k(x).$$

Our model takes as its prompt a set of closed-loop transitions $(x, u, \dot x, \text{cost})$ recorded under $\pi_k$, together with query states. It outputs $V_k$ at the query states and an estimate of $G(x)^\top \nabla V_k(x)$ there, which is all the improvement step needs.

The claim is threefold. Such a model, pretrained with physics-informed residual losses on randomly generated system families where $f$ and $G$ are known, can be deployed on held-out systems where only trajectories are available. It reaches closed-loop cost and stability comparable to per-instance physics-informed policy iteration at roughly three orders of magnitude less wall-clock time. And the iterative structure is what makes this work: refreshing the prompt with rollouts from each new controller corrects the error that a single amortized pass would lock in.

Relation to Prior Work

Two papers motivate the gap by showing that policy iteration works in high dimensions but is costly.

Physics-Informed Neural Network Policy Iteration solves generalized HJB equations with deep networks up to 12 dimensions and stabilizes a 3D quadrotor. It retrains at every iteration of every system, requires exact dynamics, and finds that its fast least-squares variant (Extreme Learning Machines, single-hidden-layer networks with random fixed input weights) collapses beyond $d=4$. That paper also supplies part of our method: its local anchor matching the value Hessian to the Riccati solution of the linearized plant. It also constrains our design. It shows near-zero equation residuals can coexist with an unstable closed loop, so we evaluate stability directly rather than residual error.

Solving nonconvex Hamilton-Jacobi-Isaacs equations with PINN-based policy iteration extends policy iteration to two-player games up to 10 dimensions. It shows that decomposing the problem into linear sub-problems beats minimizing the nonlinear equation directly by about an order of magnitude. This is our main evidence that the linear evaluation step is the right target to amortize.

The closest existing work is Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learning. It trains a physics-informed DeepONet (a network that encodes an input function and evaluates the output function at query coordinates) mapping terminal costs to value functions inside a policy iteration loop, validated on 10D LQR. Its dynamics are frozen in the weights, so any change of plant means retraining. Our model conditions on the plant itself, through trajectories, and targets infinite-horizon problems.

Solving Optimal Execution Problems via In-Context Operator Networks uses an in-context network to infer unknown dynamics from five prompt trajectories. It then spends about 20,000 policy-gradient steps on a scalar control problem. We instead put the in-context model inside the dynamic-programming loop, so the improvement step is analytic.

In-Context Policy Iteration runs policy iteration with a frozen language model over discrete, text-encoded states. It shares our loop structure but cannot represent continuous high-dimensional value functions.

The architecture comes from in-context operator learning. ICON supplies the prompt-as-examples design and the encoder/cross-attention decoder, reporting about 2% average relative error with five examples on 1D/2D problems. DeepOSets supplies permutation-invariant pooling over prompt pairs with cost linear in prompt size. That property is what we need, because our prompts are thousands of scattered samples in $\mathbb{R}^d$ rather than grid functions.

For the dimensional scaling, the relevant precedents are the HJB-PMP network of Onken et al., which shows value networks whose size grows linearly with dimension up to 150 dimensions, and CINOC, which shows operator policies on nonlinear PDEs. Both are trained per system.

Research Program

Model. The model is a set encoder over prompt transitions. Each transition is embedded as a token, and the tokens are pooled with a DeepOSets-style aggregator plus a small number of cross-attention layers into a latent "operator code." A decoder takes this code and a query state. It outputs a scalar $V_k(x)$ and a $m$-vector $h_k(x)$ (where $m$ is the control dimension) that estimates $G(x)^\top \nabla V_k(x)$. We predict $h_k$ explicitly because $G$ is not observed at test time. Using the transitions, the model can infer how controls move the state, but only through the prompt.

Training. Pretraining uses synthetic families where $f$, $G$, $\ell$, and $R$ are known and randomly drawn. The families are random LQR systems, chains and networks of coupled nonlinear pendula and oscillators with random masses and couplings, polynomial systems with random sparse coefficients, and quadrotor-like rigid-body models with random inertia. For each drawn system we generate a sequence of controllers by running policy iteration with the model in the loop, and the prompts come from those controllers' own rollouts. The loss has three parts. The first is the generalized HJB residual at query states, computed with the true $f$ and $G$, which are available only during training. The second is the consistency loss $h_k = G^\top \nabla_x V_k$. The third is the Riccati Hessian anchor near equilibrium. Training on the model's own iterates exposes it to the state-distribution shift that each controller update causes, which is a known failure point of policy iteration.

Deployment. The initial controller $\pi_0$ is LQR computed on a least-squares linearization fitted from an exploratory batch. This addresses the requirement that policy iteration start from a stabilizing controller, and it also serves as our naive baseline. Each iteration rolls out $\pi_k$ for a few seconds of simulated time, feeds the fresh transitions to the model, and sets $\pi_{k+1}$ in closed form. As a cheap safeguard, the new controller is accepted only if the predicted value decreases along the newly observed closed-loop trajectories and the average return does not rise. Policy iteration's monotone improvement property makes this a natural check. It is an empirical certificate, not a formal proof, and we will measure how often it wrongly accepts an unstable controller.

Analysis. We will measure how error scales with prompt size and dimension. We will also study how far out of the pretraining family the model degrades, for example with unseen nonlinearity types or larger coupling strengths.

Baselines

The naive reference is LQR on a linearization fitted to the same exploratory data, with no iteration. This is also $\pi_0$. Any gain over it measures what nonlinear policy iteration adds. On near-linear systems we expect it to be hard to beat, and that is informative.

The strongest accuracy competitor is per-instance PINN policy iteration as in Physics-Informed Neural Network Policy Iteration. It is run with the true dynamics (an advantage our method does not get), the same number of iterations, the same cost functions, and the same anchor loss, with its per-iteration training budget tuned on validation systems. The file reports no numerical closed-loop cost for its 12D synthetic benchmark, only that error stays "steady" up to 12 dimensions and that the quadrotor stabilizes "in under two seconds." We therefore rerun it ourselves. Beating it means our method gets within 10% of its closed-loop cost with the same stabilization rate, using at least 100 times less wall-clock time per new system. Matching its accuracy outright is not the target.

The compute-matched competitor is warm-started PINN policy iteration: a few gradient steps per iteration, initialized from the previous iterate's weights (and from a network pretrained on the same family), with the total wall-clock budget set equal to our model's. This tests whether cheap fine-tuning already gives the amortization we claim.

The structure competitor is in-context system identification followed by a solver. An ICON-style model pretrained on the same systems infers the dynamics from the same prompts, as in the optimal-execution pipeline, and per-instance PINN policy iteration is then run on the inferred surrogate.

For the game extension we compare against the PINN policy iteration solver of the HJI paper. It reports relative error $1.17\times10^{-2}$ on the 5D publisher-subscriber game, versus $1.12\times10^{-1}$ for a direct PINN. For the 10D game the file reports only convergence, and no reported numerical result was found on Lacuna for it.

We also report the PI-DeepONet of Lee and Kim, retrained per dynamics, on the LQR family. No numerical 10D error is reported in the file, only that it "successfully solved" the problem.

Two ablations isolate the key ingredients. The one-shot amortized HJB ablation uses the same architecture trained to output the optimal value directly from $\pi_0$ rollouts, with no loop. The frozen-prompt ablation keeps the loop but reuses the $\pi_0$ rollouts at every iteration, so the prompt is never refreshed under the new controller.

Why not just…?

Why not just identify the dynamics in context and hand them to an existing solver? This is the most serious threat, because a good dynamics surrogate plus per-instance policy iteration could match or exceed our accuracy. Its weakness is cost and error compounding. It still pays full per-instance training at every iteration, and HJB solvers exploit surrogate errors in rarely visited regions. The structure-competitor baseline settles this directly. If it wins on accuracy at similar cost, our claim reduces to a speed claim, and we will report it that way.

Why not just warm-start PINN policy iteration instead of pretraining an in-context model? Consecutive controllers are close, so a few gradient steps may suffice. That covers amortization across iterations but not across systems. Each new plant still starts from an unrelated network, and the file notes these multi-step loops carry high wall-clock latency. The compute-matched baseline measures exactly where the two curves cross.

Why not just train one model to map trajectories to the optimal value function directly? It is simpler and has one forward pass instead of several. The evidence against it is that directly minimizing the nonlinear equation is about ten times worse than policy iteration in the HJI study. A one-shot model also sees only states visited by $\pi_0$ and cannot correct itself where the optimal controller goes. The one-shot and frozen-prompt ablations test both points.

Why not just use nonlinear MPC? It needs no pretraining and handles constraints. But it re-solves an optimization at every step. The file reports 592 ms per step for nonlinear MPC in CINOC and a 400–600x latency penalty for nonlinear programming in Onken et al., and it gives no global feedback law. We include it as a closed-loop cost reference on the 6–12D systems and report its per-step latency alongside.

Proposed Experiments

Systems are drawn from held-out members of the pretraining families at $d \in \{4, 8, 12, 20, 30\}$. We also use the PINN policy iteration benchmark systems (inverted pendulum, 3D quadrotor, controlled Lorenz, the 12D synthetic system) held out entirely from pretraining. The 5D and 10D publisher-subscriber games serve as a two-player extension; there the evaluation step is still linear once both players' controllers are fixed.

On LQR families the exact Riccati solution gives ground-truth value error. On nonlinear systems, the reference is per-instance PINN policy iteration trained to convergence with true dynamics.

The metrics are relative value error at held-out states, closed-loop cost from a fixed set of initial conditions, the fraction of initial conditions stabilized within a horizon, how often the empirical safeguard accepts an unstable controller, and wall-clock time per new system. Stabilization rate is treated as a primary metric rather than a secondary one, following the residual-versus-stability failure documented by Meng et al..

Three checks decide the outcome. First, on held-out 12D nonlinear systems, the in-context loop must come within 10% of per-instance closed-loop cost at 100x or more lower wall-clock. Second, the refreshed-prompt loop must clearly beat both the frozen-prompt and one-shot ablations, which would show that iteration, not just amortization, carries the gain. Third, the value error must degrade gracefully, not collapse, from 12 to 30 dimensions at a fixed prompt size, a point where the least-squares approach already failed beyond 4. If the first check passes but the second fails, the right conclusion is that in-context one-shot HJB solving suffices, and the policy iteration framing should be dropped.