Neural solvers for the Hamilton-Jacobi-Bellman (HJB) equation can compute safe and optimal controllers in dimensions where grids are hopeless. The HJB equation is the partial differential equation whose solution, the value function, gives the best achievable cost from every state. These solvers train a network so that the equation's leftover error, the "residual," is near zero at randomly sampled states. They need no labeled solutions, only the system's equations of motion. The catch is that every new system, obstacle layout, or capture radius is solved from scratch. DeepReach spends 16 to 25 GPU-hours per 9D–10D problem, and the epigraph safe-control work needs 400k epochs for a 20D task.
Compact language models face a similar cost problem, and they solve it a different way. A broadly pretrained base is cheaply specialized to a narrow domain, often through small add-on parameters. The specialized model then beats much larger generalists, as AutoTriton, VeriGen and LLMLight show. We propose importing that recipe into control. First, pretrain one smooth value network across a broad family of control problems. Then specialize it to a new target system using only the HJB residual of that target as the training signal. The specialization lives in low-rank weight updates (LoRA: two thin trainable matrices added to each frozen weight matrix). There are no demonstrations, no labels and no rollouts. The claim is that this reaches from-scratch accuracy at a fraction of the compute. It should also handle targets that lie outside the family a parameter-conditioned network can interpolate over. Because the base stays frozen, one base can serve many specialists without forgetting.
Core Claim
Residual minimization from scratch is slow for two reasons. The network has to discover the gross shape of the value function, and the squared residual is a nonconvex objective that can settle on wrong solutions. A base pretrained on related systems already encodes that gross shape. Our hypothesis is that adapting it with a small number of parameters under the target's residual alone does three things. It converges faster. It ends in the correct solution more often across random seeds, because it starts near the right basin and is constrained to stay nearby. It also leaves the base intact for other targets.
Concretely, let $\theta_0$ be the frozen pretrained weights. Let $B$ and $A$ be the low-rank factors added to each weight matrix. Let $\mathcal{R}_\theta(s,t)$ be the target system's HJB residual at state $s$ and time $t$, and let $g$ be the target's terminal cost. Specialization then solves
$$\min_{A,B}\ \mathbb{E}_{(s,t)}\big[\mathcal{R}_{\theta_0+BA}(s,t)^2\big] + \lambda\,\mathbb{E}_{s}\big[(V_{\theta_0+BA}(s,T)-g(s))^2\big],$$
where $T$ is the horizon and $\lambda$ weights the terminal condition. Nothing in this objective requires solved examples of the target.
Relation to Prior Work
Two papers amortize HJB solutions across tasks, and they set the bar we must clear. Hamilton-Jacobi policy iteration via deep operator learning trains an operator network that maps terminal costs to value functions zero-shot in up to 10D. It covers only variation in the cost under fixed control-affine dynamics, and it has no mechanism for adapting when a query falls outside that family. Meta value learning meta-trains a value network with the HJB equation as a regularizer. It then fine-tunes all weights online on optimal trajectories produced by a boundary-value-problem (BVP) solver. That design has three limits: it was shown only on a 3D unicycle, it diverges when the BVP solver's initial guess is poor, and it needs supervised samples at adaptation time. Our proposal differs from both. Adaptation uses the residual only, which removes the BVP dependence. It touches a small parameter subset. And it targets structural changes, such as new dynamics coefficients, obstacle topology or capture geometry, that conditioning cannot reach.
The specialization literature motivates the recipe and also warns about it. Specializing smaller language models towards multi-step reasoning documents a specialization tax: math-specialized models lose substantial accuracy on BigBench Hard. This is why we measure forgetting explicitly and keep the base frozen. DiffPDE and AutoTriton specialize models against execution-based verifiers. We play the same game with a continuous verifier: the residual is exact, available at every sampled state, and differentiable.
The HJB literature constrains the architecture. DeepReach shows that ReLU networks give poor gradients for this problem and that sine-activation networks work. Continuous-time value iteration for multi-agent RL similarly needs smooth activations. Parameterized optimal control needed smooth residual networks for stable value gradients. We therefore keep a sine-activation base rather than a transformer. In a sine network, low-rank updates to the weights effectively shift frequencies, and whether that is benign is itself a question we test.
Research Program
We build a pretraining family around each benchmark. For Air3D-style collision avoidance, the family varies the vehicles' speeds, turn-rate limits and capture radius. For narrow-passage and multi-vehicle driving, it varies the geometry and the number of interacting vehicles' parameters. For the 2D boat, 8D pursuer-evader and 20D multi-agent navigation tasks of the epigraph framework, it varies drift fields and obstacle placement. The base takes the family parameters as an extra input, so it is a conditioned, amortized network. We do this deliberately: the conditioned base is also our strongest zero-shot baseline, and adaptation is layered on top of it.
Targets are split three ways. Some lie inside the family. Some lie outside it in the parameter range, such as a larger radius or a speed ratio never seen. Some are structurally new, such as non-circular capture sets, a new obstacle topology, or an added drift term.
We analyze three things. The first is time-to-accuracy: GPU time to reach a fixed residual and ground-truth error, as a function of adapter rank. The second is solution correctness across seeds. Here we ask whether warm-started adaptation lands in the wrong solution less often than from-scratch training, which we can check against grid ground truth in 3D. The third is the specialization tax: whether the family-level residual degrades after adaptation, which is zero by construction for frozen-base adapters but not for full fine-tuning.
Baselines
The naive reference points are the pretrained conditioned base used zero-shot, with no adaptation, and a sine network trained from scratch on the target under the DeepReach recipe. The zero-shot base measures how much amortization alone buys. From-scratch training is the cost we aim to avoid.
The strongest existing method on accuracy is full DeepReach training per target. On Air3D, where ground truth exists, DeepReach reports a mean squared error of 1.01×10⁻⁴ and a volume error of 0.43%, with 16 to 25 hours of training at 9D–10D. Beating it means matching those errors on Air3D variants within a fixed fraction of its GPU time. We set the target at one tenth in advance and declare it before running. For higher-dimensional tasks, beating it means matching its safety-violation rate under rollouts.
For safe navigation, the reference is the epigraph physics-informed controller, which reports a 100% safety rate on 2D boat and 20D multi-agent navigation. Here we must preserve 100% safety, after the same conformal wrapper, at much lower training cost.
For amortization, we run an operator-network baseline in the style of deep-operator policy iteration on the same families. We also run meta value learning on its unicycle suite, where it reports costs 4–5% above the minimum and 90% success within about 7 seconds of adaptation. No reported result was found on Lacuna for the operator method on the reachability or epigraph benchmarks, so we will run it ourselves.
Two ablations isolate the key ingredients. Adapters on a randomly initialized base isolate the value of pretraining. Full fine-tuning of the pretrained base isolates the value of the low-rank restriction.
All methods share the same collocation sampler, curriculum over time horizons, GPU type and wall-clock accounting. Pretraining cost is reported separately and amortized over the number of targets served. Each method gets the same number of hyperparameter configurations.
Why not just…?
Why not just condition one network on the system parameters and skip adaptation? For in-family targets this may be enough, and the in-family split will show it. If adaptation adds nothing there, we say so. The operator approach of deep-operator policy iteration is demonstrated only for variation in quadratic cost under fixed dynamics. A conditioning input cannot represent a structural change it was never given a coordinate for. The out-of-range and structural splits are where we expect the zero-shot base to fail and residual-only adaptation to recover. If it does not fail there, the proposal's main claim is wrong.
Why not just warm-start and fine-tune all the weights? This is the most serious threat to the low-rank component, and it may match adapters on speed and accuracy. We run it head to head. The adapter design keeps two advantages that full fine-tuning lacks even if accuracy ties. There is no forgetting of the family, which we measure as the family-level residual after adaptation. And storage for many specialists is a small fraction of the base. The adapters' further advantage, fewer wrong-basin solutions across seeds, is a hypothesis rather than a guarantee, and the Air3D seed study decides it. If full fine-tuning wins on all three counts, the paper reports that pretraining, not low rank, is the ingredient that matters.
Why not just meta-learn the initialization as in meta value learning? Meta value learning needs supervised BVP trajectories at adaptation time, and its authors report divergence when a query falls outside the training distribution. It was also shown only in 3D. Our adaptation needs only the target's equations. We compare on the authors' unicycle suite and then on the higher-dimensional tasks where their pipeline was never run.
Why not just specialize with model-free RL or imitation? On a roughly 1,000-dimensional advection-diffusion control problem, parameterized optimal control reports a sub-optimality of 0.03 for the HJB-based method. PPO and TD3, two standard model-free RL algorithms, stayed above 0.20 after 15,000 simulations, needed 20x more simulations, and sometimes diverged. HJBPPO found that plain PPO converged to the HJB solution in only 4 of 10 MuJoCo tasks. Imitation would require exactly the solved trajectories that residual training avoids. We therefore do not rerun RL, except as a sanity check on the 2D boat task.
Proposed Experiments
The decisive experiment runs on Air3D variants, where grid ground truth exists. For each held-out target and each of several seeds, we compare four methods: from-scratch training, zero-shot base, full fine-tuning, and low-rank adaptation at ranks from 1 to 32. We plot mean squared error and volume error against GPU time. The proposal succeeds if three things hold. Adaptation reaches DeepReach's reported error level within one tenth of its compute on out-of-family targets. The zero-shot base misses that error level on those targets. And the spread of wrong-basin outcomes across seeds is narrower than from scratch.
The scaling experiment repeats the comparison on the 9D three-vehicle and 10D narrow-passage problems, using rollout safety violations. It also covers the 8D pursuer-evader and 20D multi-agent navigation problems, where success means keeping the conformally certified 100% safety rate and the reported cost advantage.
A final check probes the architecture tension. We adapt only first-layer frequencies versus only later layers. This tells us whether low-rank updates in sine networks preserve gradient quality, and it shows where target-specific structure lives in a pretrained value model.
References
- DeepReach: A Deep Learning Approach to High-Dimensional Reachability
- A Physics-Informed Machine Learning Framework for Safe and Optimal Control of Autonomous Systems
- AUTOTRITON: Automatic Triton Programming with Reinforcement Learning in LLMs
- VeriGen: A Large Language Model for Verilog Code Generation
- LLMLight: Large Language Models as Traffic Signal Control Agents
- Hamilton-Jacobi based policy-iteration via deep operator learning
- Meta Value Learning for Fast Policy-Centric Optimal Motion Planning
- Specializing Smaller Language Models towards Multi-Step Reasoning
- DiffPDE: Masked Diffusion Language Models as PDE Solver
- Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning
- Neural network approaches for parameterized optimal control
- Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO)