Feed-Forward Reachability: Predicting Safe Sets Directly from Gaussian Scene Maps

Robots that must avoid crashing face two expensive steps. They have to build a map of a new place, and then they have to compute, for that map, the set of states (positions and velocities, headings, steering angles) from which a crash can still be avoided. Computer vision has largely removed the cost of the first step. Networks such as GS-LRM and DepthSplat produce a 3D map made of small ellipsoids, called 3D Gaussians, in well under a second, with no slow per-scene fitting. The second step has not had the same change. Hamilton-Jacobi reachability is the standard tool here: it solves a partial differential equation whose solution, a "value function," is negative exactly on the unrecoverable states. Its most scalable neural solver, DeepReach, still needs 16 to 25 GPU-hours per scenario. Reachability therefore has its own per-scene optimization overhead, on top of the well-known exponential blowup with state dimension.

We propose to treat the second step the way vision treated the first. We would train once, over a large distribution of scenes, a neural solution operator: a network that reads a set of Gaussian ellipsoids around the robot and outputs the reachability value function for a fixed high-dimensional vehicle model in one forward pass. Training uses the same physics-based residual loss DeepReach uses, so it needs no ground-truth value functions. One structural fact makes this tractable. Over a finite safety horizon $T$, the robot can only touch obstacles within a bounded distance, so the operator only ever sees a local, permutation-invariant set of ellipsoids, never a whole map. In low dimensions, a fast local grid repair can then certify the predicted value function exactly. In high dimensions, a calibrated check bounds how often it wrongly labels states as safe.

Core Claim

Fix a vehicle model $\dot x = f(x,u,d)$ with state $x\in\mathbb{R}^n$, control $u$ and disturbance $d$. For that fixed model, the per-scene reachability problem can be amortized by a network $V_\theta(x,t;\mathcal{G})$, where $\mathcal{G}=\{(\mu_i,\Sigma_i)\}$ is the set of Gaussian means and covariances within radius $R = v_{\max}T + r$ of the robot ($v_{\max}$ the maximum speed, $r$ the robot radius). The network is trained across procedurally generated and reconstructed scenes by minimizing the Hamilton-Jacobi-Isaacs variational-inequality residual:

$$ \mathcal{L}(\theta)=\mathbb{E}_{\mathcal{G},x,t}\Big|\min\Big\{\partial_t V_\theta + H(x,\nabla_x V_\theta),\; l_{\mathcal{G}}(x)-V_\theta\Big\}\Big|, $$

Here $H$ is the max-over-controls, min-over-disturbances Hamiltonian, and $l_{\mathcal{G}}(x)$ is the analytic signed distance from the robot body to the nearest ellipsoid, computed as in SAFER-Splat.

The claim has three parts, and each can be tested. First, the operator's safe set matches per-scene solvers on unseen scenes. Second, it does so in milliseconds rather than hours. Third, it is meaningfully less conservative than the obvious shortcut of separating vehicle dynamics from obstacle geometry.

Positioning Against Prior Work

The closest relatives are operator-learning methods for Hamilton-Jacobi equations. Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learning maps terminal cost functions to value functions in 10D linear-quadratic problems. The Pontryagin neural operator generalizes across a 2D family of collision-zone parameters in a 5D game. Both supply our method, a solution operator trained from physics residuals. Both also constrain our expectations. The first only handles smooth quadratic costs. The second only varies two scalar parameters. Neither faces sharp, many-obstacle collision boundaries or perceived geometry. Our input is a variable-size set of ellipsoids, and that changes the encoder design and the difficulty of the problem.

DeepReach motivates the work. It shows that neural residual training reaches 9D and 10D. It also shows that each new scenario costs a full day, which is exactly the overhead we amortize away.

SAFER-Splat supplies the geometric interface: ellipsoid distance queries and pruning of distant Gaussians. It also motivates the dynamics gap, because its barrier uses only position distance and its map is still fit online by gradient descent.

SAGE learns a reachability critic on a 32D autoencoder latent. It serves as our counterpoint on representation. Its latent discards metric geometry, and the authors report perception taking up to 60% of latency. We keep explicit ellipsoids so the target function $l_{\mathcal{G}}$ stays exact.

Refining Almost-Safe Value Functions on the Fly supplies the certification step. Its local grid repair turns an approximately correct value function into a certified one in real time, but it is limited to roughly 4 to 6 dimensions and depends on a good initial guess. Our operator provides that guess.

Research Program

We would build three pieces.

The first is an operator architecture. A set encoder (a transformer over ellipsoid tokens, expressed in the robot's frame and including the $\Sigma_i$ shape terms) feeds a SIREN-style trunk over $(x,t)$, following DeepReach's sinusoidal network and backward-in-time curriculum. Locality in the robot's frame is the key design choice. It caps the input size and makes the operator translation-invariant. Training scenes come from two sources: procedural ellipsoid clutter, including narrow passages, and floor-level obstacle slices taken from ScanNet and DL3DV scenes reconstructed by DepthSplat. Mixing these two sources exposes the operator to the floaters and imperfect boundaries that feed-forward reconstruction actually produces.

The second is a verification layer, which differs by dimension. For a 4D extended-bicycle ground robot (the Jackal model used by HJ-Patch), we pass $V_\theta$ to HJ-Patch and measure how much repair is needed and how long it takes. For higher-dimensional models, where grids are impossible, we calibrate a level-set shift on held-out scenes. We draw states, roll out the operator's safety controller under sampled worst-case disturbances, and choose the shift so that the false-safe rate stays below a target with conformal coverage. This is in the spirit of ReCORS. We would state plainly that this is a statistical guarantee, not a formal one.

The third is a reconstruction-robustness study. For the same scene, we compare safe sets computed from ground-truth meshes and from DepthSplat Gaussians, and we measure how much margin inflation absorbs reconstruction error.

Baselines

The naive reference is a distance-only barrier function evaluated on feed-forward Gaussians. This is SAFER-Splat's quadratic-program filter fed by DepthSplat instead of online fitting. It is what most practitioners would build first. We run it with the identical Gaussian maps and tune its class-K gain on the same validation scenes. Beating it means fewer collisions at equal or higher speed in high-speed and narrow-passage scenes, where position-only margins ignore braking distance.

The accuracy gold standard is DeepReach trained per scene. On the 3D Air3D benchmark, DeepReach reports a mean squared error of $1.01\times10^{-4}$ and a 0.43% volume error against the exact grid solver, at 16 to 25 GPU-hours per 9D/10D configuration. On 3D and 4D scenes, where grid ground truth exists, we compare safe-set intersection-over-union and false-safe rate with both methods given the same network size. Success means coming within a small factor of DeepReach's volume error at roughly a millionth of the compute per scene.

The strongest practical competitor is Parametric FaSTrack. It precomputes, once, an obstacle-independent tracking-error bubble for 6D and 13D models, then plans around obstacles inflated by that bubble. It reports 100% success over 20 runs against 65% for sampling-based model-predictive path-integral control, and 40% faster navigation than static FaSTrack. We run it with the same Gaussian obstacles and the same dynamics, using its released training protocol. Beating it requires equal safety with measurably larger safe sets and higher average speed in clutter.

The decisive ablations are three. We replace ellipsoid tokens with a SAGE-like learned image latent. We remove locality by encoding the whole map. We swap the residual-trained operator for a supervised one trained on grid solutions, in low dimensions only.

Why not just…?

Why not just use Parametric FaSTrack, which already avoids per-scene reachability? This is the real threat. Separating tracking error from geometry is exact only up to the conservatism of a geometry-blind bubble. Where obstacle shape and vehicle momentum interact, as in narrow passages entered at speed or corners that demand early braking, the bubble must be sized for the worst case everywhere. The comparison above settles the question directly. If its safe-set volume and speed match ours in cluttered scenes, the operator is not worth its training cost, and we would report that.

Why not just warm-start DeepReach or HJ-Patch per scene? HJ-Patch is grid-bound, which is why we use it only as a certifier in 4D. Briefly fine-tuning a pretrained DeepReach per scene is a fair alternative, and we include it on an accuracy-versus-wall-clock curve. If 30 seconds of fine-tuning matches the operator, the right conclusion is that the operator should serve as the initializer, and we would test that hybrid rather than argue against it.

Why not just use a distance barrier on fast feed-forward maps? This removes the mapping overhead but not the dynamics gap. FaSTrack and HJ-Patch both report failures when position-only safety ignores momentum: in the HJ-Patch Jackal hardware tests, the naive baselines collided in 5 of 5 runs, while HJ-Patch collided in 1 of 10 and refineCBF in 0 of 10. The naive baseline above quantifies how large that gap is in our scenes.

Proposed Experiments

In 3D Dubins and 4D bicycle models across 1,000 held-out reconstructed scenes, we measure safe-set intersection-over-union and false-safe rate against grid truth, plus inference latency. We also measure the fraction of the boundary HJ-Patch must repair and its CPU time; the Jackal hardware reference is 1 of 10 collisions for HJ-Patch and 0 of 10 for refineCBF.

For 6D Dubins and 13D quadcopter models, we run Parametric FaSTrack's obstacle courses with obstacles drawn as Gaussians, reporting success rate over 20 runs, average speed, and calibrated false-safe rate.

For a vision-in-the-loop test, we use Learn-to-Race Thruxton with DepthSplat-reconstructed track boundaries. We report completion and average adjusted track speed against SAGE (reported above 80% completion early in training) and the benchmark's SAC baseline (31% completion, 50.5 km/h).

The decisive checks are three. The operator's false-safe rate on unseen reconstructed scenes must stay below a pre-registered threshold after calibration. Its safe set must be strictly larger than Parametric FaSTrack's in narrow-passage scenes. The ellipsoid-token ablation must beat the latent ablation on generalization to ScanNet-derived layouts not seen in training. Failing the second check would refute the proposal's reason to exist.