Is the Eikonal Value Prior Optimizer-Limited? Preconditioning Physics-Informed Offline RL

Offline goal-conditioned reinforcement learning trains an agent to reach arbitrary goals using only a fixed dataset of past trajectories, with no new interaction. A recent result showed that a small piece of physics can make this far more data-efficient. The idea is to require the learned "distance-to-goal" value function to obey the Eikonal equation, the equation describing a wavefront spreading at fixed speed. A value function that obeys it behaves like a true travel-distance map, so it bends around walls instead of cutting straight through them. On the hardest ant-navigation maze, this raised success from 7% to 79% (Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning).

The scientific-computing literature carries a warning about this approach. Putting a derivative-based penalty into a neural network's loss tends to make the loss surface badly conditioned: steep in a few directions and nearly flat in others, so ordinary optimizers either bounce or stall. This note proposes to test whether Eikonal value learning is limited by that optimization difficulty rather than by the physics prior itself, and to fix it if so. The test asks two things. Does the curvature of the value loss degrade as the Eikonal weight and the dataset size grow? Does a cheap curvature-corrected update, applied only to the value network, turn the same data into higher success rates and keep improving as data is added? The same experiments answer a question the original paper left open: Eikonal regularization gives little benefit on contact-rich tasks, and we do not yet know whether the prior is wrong there or merely hard to optimize.

Core Claim

We claim that the Eikonal-regularized value objective inherits the ill-conditioning documented for physics-informed neural networks. We further claim that a Gauss-Newton update, which corrects for the curvature of the joint Bellman-plus-Eikonal residual using a low-rank sketch, will produce three effects. First, it will raise success on long-horizon navigation at fixed data. Second, it will make success increase monotonically with dataset size where first-order training saturates or regresses. Third, it will separate the two possible causes of failure on contact-rich tasks. The claim is falsifiable on each count: if tuning the Eikonal weight alone with Adam matches the preconditioned method, the optimization-limited hypothesis is wrong.

Why This Should Be True, and What Constrains It

Three PINN results motivate the hypothesis. Here a PINN, or physics-informed neural network, is a network trained to satisfy a differential equation through a penalty term. Challenges in Training PINNs: A Loss Landscape Perspective analyzes the loss curvature (Hessian) spectra and finds that the differential-operator term, not the boundary data, creates huge outlier eigenvalues. It also reports that conditioning worsens as the number of sampled points grows. Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks shows that gradients from the equation residual can exceed those from the data-fitting term by orders of magnitude. The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks shows that this curvature does not disappear with width in nonlinear equations. The Eikonal equation is nonlinear, because it constrains the gradient norm.

In Eik-HIQL, the Bellman/expectile target plays the role of boundary data and the Eikonal penalty plays the role of the interior residual. Eik-HIQL trains both with plain first-order steps and reports no diagnostics. Two further findings in that paper are suggestive. A constant-speed profile often beat more detailed wall maps, and gains vanished on contact tasks. Both are consistent with an optimizer that cannot exploit a sharper prior, though neither proves it.

Two lines of work supply the method. Fast Convergence of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks proves that curvature-corrected steps converge at a rate independent of the condition number. On 2D Helmholtz it reaches $6.67 \times 10^{-3}$ error where Adam, SGD, and L-BFGS hover near 1.0. Its exact Gram-matrix inversion costs $O(n^3)$, which is useless at offline-RL dataset sizes. The NysNewton-CG method of Rathore et al. replaces exact inversion with a randomized low-rank (Nyström) preconditioner. It is still over 300 times slower per iteration than L-BFGS on the wave equation, so it constrains our design: the full solve cannot run at every step. Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (ACKTR) shows that preconditioning the critic's least-squares loss with a Gauss-Newton approximation pays off in RL. It reports 2–3x sample efficiency at 10–25% overhead. ACKTR is online, on-policy, and has no derivative terms in its critic loss.

The contrast with prior work is therefore specific. ACKTR preconditions a Bellman loss without physics. The PINN natural-gradient methods precondition physics losses without RL. Eik-HIQL combines physics and RL without preconditioning. No paper in the file measures or corrects the curvature of a physics-regularized RL value loss.

Research Program

Write the value network as $V_\theta(s,g)$, with parameters $\theta$, state $s$, and goal $g$. On a minibatch we stack two residual vectors. The first is the HIQL expectile-Bellman residual $r_B(\theta)$, the per-sample gap to the bootstrapped target, scaled by the expectile weight. The second is the Eikonal residual $r_E(\theta)$, with entries $\|\nabla_s V_\theta(s,g)\| - 1/S(s)$ at sampled states, where $S$ is the speed profile ($S=1$ by default). The loss is

$$L(\theta) = \tfrac{1}{2}\|r_B(\theta)\|^2 + \tfrac{\lambda}{2}\|r_E(\theta)\|^2,$$

with Eikonal weight $\lambda$. Let $J_B$ and $J_E$ be the Jacobians of the two residuals with respect to $\theta$; $J_E$ needs mixed input-parameter derivatives, available by double backpropagation. The proposed update is a damped Gauss-Newton step,

$$\Delta\theta = -\left(J_B^\top J_B + \lambda J_E^\top J_E + \mu I\right)^{-1} \nabla_\theta L,$$

where $\mu$ is a damping constant. We solve this approximately with a few conjugate-gradient iterations, preconditioned by a randomized Nyström sketch of the bracketed matrix. The sketch is refreshed only every $k$ steps and reused in between, and Adam steps run in between. Only the value network gets this treatment; HIQL's high- and low-level policies keep their original first-order training, as in Eik-HIQL. The step is the least-squares analogue of the PINN natural gradient. ACKTR's critic step is the special case $\lambda = 0$.

The program runs in three stages. The first is purely diagnostic and needs no new method. On OGBench navigation and contact tasks we train Eik-HIQL as published. Throughout training we record the top Hessian eigenvalues of $L$ by Lanczos iteration, an estimate of the condition number from the Nyström sketch, and the per-layer ratio of Bellman to Eikonal gradient norms in the style of Wang et al. We sweep $\lambda$ and the fraction of the dataset used (10%, 25%, 50%, 100%). The prediction from the PINN literature is that conditioning worsens with both. The second stage adds the preconditioned value update and repeats the sweep. The third stage applies the diagnostics to contact-rich tasks to decide which failure mode applies there.

Two terms need care. "Sample efficiency" here means success as a function of how much offline data the agent is given, which is the quantity the OGBench coverage results show to be decisive (OGBench: Benchmarking Offline Goal-Conditioned RL). The analogy to the PINN "more points, worse conditioning" result is a hypothesis, not an established fact. Offline states are not a uniform collocation grid, and the experiment exists to check whether the effect transfers.

Baselines

The naive reference point is plain HIQL without the Eikonal term, trained with Adam using OGBench defaults. It tells us how much any physics prior contributes. The strongest existing method is Eik-HIQL itself as published (Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning). It reports 79% success on Antmaze-Giant against 7% for HIQL. The file does not say which dataset variant that figure comes from, so we will first reproduce it on the OGBench navigate and stitch variants and treat our reproduction as the reference. Beating it means a statistically reliable gain in success rate over at least eight seeds on Antmaze-Giant and HumanoidMaze-Giant at equal wall-clock training time. HumanoidMaze-Giant is a task where OGBench reports 0% for all tested methods; the file gives no reported Eik-HIQL number for it.

The fair-comparison rule is strict. Every arm uses the same datasets, the same network architecture, and the same training wall-clock budget, measured on identical hardware. Every arm gets the same number of hyperparameter trials, 20 random-search configurations. For Adam arms the search covers learning rate and $\lambda$. For the preconditioned arm it covers $\lambda$, damping $\mu$, sketch rank, and refresh interval $k$. Equalizing wall-clock rather than step count matters because the curvature step is more expensive.

A tuned-weight Adam baseline gets the full search budget over $\lambda$ and learning rate, including a schedule that ramps $\lambda$ up over training, following the curriculum idea of Characterizing possible failure modes in physics-informed neural networks. An adaptive-weighting baseline uses ReLoBRaLo (Multi-Objective Loss Balancing for Physics-Informed Deep Learning) to rebalance the Bellman and Eikonal terms from their loss histories at the cost of one backward pass.

Two ablations isolate the key ingredient. The first, Bellman-only preconditioning, is ACKTR-style with $\lambda=0$ inside the curvature matrix but the Eikonal loss still in the gradient. The second preconditions the Eikonal term only. If the joint curvature matters, as we claim, only the full version should close the gap; if Bellman-only preconditioning does just as well, the benefit is ordinary critic preconditioning and has nothing to do with the physics term.

Why not just…?

Why not just tune the Eikonal weight, or anneal it, with Adam? This is the most serious threat. Nothing in the file shows that Eik-HIQL was tuned aggressively, so a better $\lambda$ schedule may recover much of the gap. Bonfanti et al. argue that weight scaling cannot remove curvature that persists in nonlinear residuals, and Wang et al. report that gradient magnitudes conflict across layers, which a single scalar weight cannot fix. Those results come from PDE solving, not RL. The tuned-weight arm with an equal search budget settles the question directly. If it matches the preconditioned method, we will report that Eikonal value learning is limited by weighting, not by curvature, and the optimization claim fails.

Why not just use adaptive loss balancing such as ReLoBRaLo? It is cheap and fixes magnitude imbalance between the two loss terms. It does not change step direction inside a narrow valley, which is the failure Rathore et al. attribute to the operator term. We will check whether ReLoBRaLo narrows the Bellman-to-Eikonal gradient-ratio diagnostic while leaving the measured condition number unchanged. If it does, and success still lags, that shows magnitude was not the bottleneck.

Why not just collect or use more data? OGBench shows that coverage, not quantity, drives success: a pick-and-place task falls from 99% to 6% without noisy data. The PINN scalability result suggests that more points can make physics-regularized training harder. The dataset-fraction sweep tests this directly. If first-order Eik-HIQL improves monotonically with data and its condition number stays flat, the scalability concern does not transfer to offline RL, and the case for preconditioning rests only on the fixed-data comparison.

Why not just use a second-order method like ACKTR on the critic? That is our Bellman-only ablation, and it is a legitimate competitor because it is proven in RL at low overhead. It ignores the curvature contributed by the derivative term. If the joint-curvature claim is right, this arm should help on plain HIQL but much less on Eik-HIQL.

Proposed Experiments

The benchmark is OGBench: PointMaze and AntMaze (large and giant) and HumanoidMaze-Giant in the navigate and stitch variants for locomotion, and AntSoccer, Cube, and Scene for contact. Metrics are goal-reaching success rate, the area under the success-versus-dataset-fraction curve, wall-clock time to reach the reproduced Eik-HIQL final success, and the curvature diagnostics described above.

There are three decisive checks. The first is a correlation test across runs: if the optimization hypothesis is right, the measured condition number of the value loss should predict final success within Eik-HIQL arms, and preconditioning should both lower it and raise success. The second is the data-scaling curve: first-order Eik-HIQL should flatten or dip as the data fraction grows on the giant mazes, while the preconditioned arm keeps improving. The third is the contact-task interpretation. If preconditioning makes the value loss well-conditioned on Cube or AntSoccer and success still does not improve, the Eikonal prior is mismatched to discontinuous contact values, and we report it as a negative result that rules out "better optimization" as the fix. If success does improve, the contact failure reported by Giammarino et al. was at least partly an optimizer artifact.

We will also report per-update overhead against the 10–25% that ACKTR reports. A method that wins only at several times the cost of Adam, under the equal-wall-clock rule, does not count as a win.