Local Gaussian-Process Residuals for Offline World Models

Model-based offline reinforcement learning trains a policy inside a learned simulator of the environment. That simulator, called a world model, is fit to a fixed log of past transitions. Its central risk is that the policy learns to steer into states the log never covered, where the model is confidently wrong. The standard guard is an ensemble of neural networks: where the members disagree, the agent's reward is penalized. The guard has a known weakness. Ensemble disagreement comes from random initialization and optimizer noise, so it can collapse. Members can agree on data they have never seen, and then the penalty goes to zero exactly where it is needed. We propose a different epistemic signal, one that grows away from the data by construction. We fit a nearest-neighbor Gaussian process over the ensemble's own learned features, using the entire offline dataset. A Gaussian process (GP) is a regression model whose predictive variance rises with distance from observed data. Its posterior variance supplies the pessimism penalty, and its posterior mean corrects the ensemble's local prediction errors.

This is now affordable because of results on the scalability side. Nearest-neighbor GPs condition each prediction on only its $m$ closest training points. They have been fit to 1.6 million points in about 30 seconds on a laptop (Leveraging Locality and Robustness to Achieve Massively Scalable Gaussian Process Regression), which is the size of a large offline control dataset. The offline setting also removes the main weakness of scalable GP methods in reinforcement learning, which is that kernel settings drift as new data arrive. Here the data never change.

Core Claim

Ensemble disagreement and actual model error come apart in the parts of state-action space that matter for offline policy optimization. A local GP residual, fit in a learned feature space, tracks model error more faithfully and does not collapse. The claim has three testable parts. The GP's posterior variance, used as a penalty, predicts true one-step error on held-out off-distribution transitions better than ensemble standard deviation does. Its posterior mean reduces one-step error near the data, because local conditioning keeps sharp features such as contact events that a low-rank sparse GP would smooth away. Together these raise normalized returns on D4RL locomotion tasks relative to ensemble-penalty methods.

Concretely, let $\bar\mu(s,a)$ be the ensemble's mean next-state prediction and $\phi(s,a)$ its low-dimensional bottleneck features. We fit a nearest-neighbor GP $g$ to the residuals $s' - \bar\mu(s,a)$ as a function of $\phi(s,a)$. The world model then predicts $\bar\mu + \mathbb{E}[g]$. Model rollouts use the penalized reward

$$\tilde r(s,a) = r(s,a) - \lambda\, \sigma_g(\phi(s,a)),$$

where $\sigma_g$ is the GP posterior standard deviation (summed over output dimensions) and $\lambda$ is the pessimism weight. This is the penalty form of MOPO, which uses the maximum ensemble standard deviation, with a different uncertainty signal in its place.

Relation to Prior Work

Two pieces of prior work motivate the proposal. Deep Ensemble as a Gaussian Process Posterior and Calibrating Deep Ensemble through Functional Variational Inference argue that ensemble diversity is a byproduct of optimizer randomness and can collapse. Their remedy is to retrain the ensemble jointly under a functional prior, and they test it only on image classification and bandits. We leave ensemble training untouched and add an uncertainty source that does not depend on member diversity. We also test in multi-step dynamics, where they did not. CBOP shows that a more principled treatment of ensemble uncertainty, via lower-confidence-bound value expansion, pays off substantially. We attack the same weakness at the level of the model's uncertainty estimate rather than the value target, so the two approaches could be combined.

Three pieces of prior work supply the method. GPnn (Leveraging Locality and Robustness…) contributes three components: estimating hyperparameters on a subset, predicting from nearest neighbors, and a post-hoc noise calibration step. Its result that the predictive mean becomes insensitive to kernel misspecification as $n$ grows is why we expect the residual correction to be safe. Exploration with Scalable Gaussian Process Reinforcement Learning used nearest-neighbor GPs (MuyGPs) for exploration bonuses in low-dimensional sparse-reward tasks. It modeled values rather than a forward world model, and used raw inputs. We use the GP for transitions, for pessimism rather than optimism, and in a learned feature space.

Other work constrains the design. Both nearest-neighbor papers report that neighbor quality degrades in high dimensions. That is why we fit the GP on a bottleneck embedding rather than on the raw inputs, which have 23 or more dimensions for HalfCheetah. Approximation-Aware Bayesian Optimization shows that inducing-point GPs, which compress the data into a fixed set of pseudo-inputs, over-smooth sharp local structure. Gadd et al. (Sample-efficient reinforcement learning using deep Gaussian processes) found single-layer sparse GPs too smooth for contact dynamics and instead used stacked GP layers, which are expensive to train. Our learned embedding plays the cheap version of their input-warping role, and local conditioning avoids the representational-budget problem.

Research Program

We train a standard seven-member probabilistic ensemble with a shared-width architecture, keeping the top five, as in MBPO. One hidden layer is narrowed to a bottleneck of 8 to 32 units, and the averaged bottleneck activations serve as $\phi$. On the full offline dataset we compute residuals and estimate Matérn kernel hyperparameters on a random subset of about $10^4$ points. We then index all $\phi$ values for approximate nearest-neighbor search. At rollout time each query conditions on its $m \in \{32, 64, 128\}$ neighbors. Queries are batched on the GPU, and each costs $O(m^3)$ regardless of dataset size. The policy learner is SAC on short branched rollouts, following MOPO and MBPO.

The analysis stage asks when disagreement and GP variance diverge, and whether the divergence happens in contact-heavy regions. The build stage adds the residual mean. We measure the cost of both in wall-clock time relative to the ensemble alone.

Baselines

The naive reference point is the plain ensemble world model with no uncertainty penalty ($\lambda = 0$, MBPO-style rollouts on offline data). It tells us how much pessimism is worth at all on each dataset.

The primary baseline is MOPO (MOPO). It is the exact method we modify: same ensemble, same rollout length, same SAC settings, with the maximum ensemble standard deviation as the penalty. It is re-run in our codebase with the same number of seeds (five) and the same $\lambda$ sweep. Beating it means higher mean normalized D4RL score across the nine HalfCheetah, Hopper, and Walker2d medium, medium-replay, and medium-expert datasets, with the gap exceeding the pooled seed standard error. The D4RL scale runs from 0 (random policy) to 100 (expert) (D4RL).

The strongest existing method in the file is CBOP (Conservative Bayesian Model-Based Value Expansion). It reports state-of-the-art results on 11 of 18 D4RL datasets and an average improvement of 116.4% over MOPO, 23.2% over MOReL and 23.7% over COMBO across medium and replay datasets. No per-task normalized scores were found on Lacuna for CBOP or MOPO. We will therefore re-run CBOP from its released settings on the same datasets and seeds, and check our MOPO reproduction against CBOP's reported relative margin. CBOP changes the value target while we change the model uncertainty, so we also run CBOP with our GP variance substituted for its ensemble variance. Beating CBOP outright is not required for the claim. Improving CBOP when our signal is plugged in would be sufficient.

A calibration-fix baseline trains the ensemble with randomized prior networks, as described in Bhouri et al.. This is the cheapest direct remedy for ensemble collapse. Where compute permits, we also train with DE-GP's functional objective (Deep Ensemble as a Gaussian Process Posterior). Each gets the same parameter count and the same training-epoch budget.

Three ablations isolate the key ingredient. The first replaces GP variance with the mean distance to the $m$ nearest neighbors in $\phi$-space, with no GP. The second fits the GP on raw state-action inputs instead of $\phi$. The third replaces the nearest-neighbor GP with a sparse variational GP (Gaussian Processes for Big Data) of matched wall-clock cost. We also toggle the residual mean on and off with the penalty held fixed. All methods share hyperparameter search budgets of equal size over $\lambda$ and rollout length.

Why not just…?

Why not just use nearest-neighbor distance in feature space as the penalty? This is the most serious threat. Local GP variance is largely a smooth function of neighbor distances. The GP adds two things a raw distance lacks: a scale calibrated in units of next-state error through the post-hoc noise calibration, which makes $\lambda$ transferable across tasks, and a residual mean that corrects predictions. The distance ablation settles the penalty question directly. If it matches GP variance on both calibration and return, we will report that. In that case the claim narrows to "feature-space locality beats disagreement," and the GP's contribution is judged by the residual-mean toggle alone.

Why not just fix the ensemble with randomized priors or functional variational training? These remedies attack collapse directly, and DE-GP improved out-of-distribution detection AUROC from 0.922 to 0.980 on static images (Calibrating Deep Ensemble through Functional Variational Inference). They still express uncertainty through member diversity, and they have not been tested on dynamics. DE-GP also couples member training, which removes the ensemble's parallelism. The calibration experiment below compares all three uncertainty sources against true error. If randomized priors close the gap, the case for the GP penalty weakens to one of cost and simplicity.

Why not just use a sparse variational GP world model instead of a residual on an ensemble? Sparse GPs smooth contact structure, according to both Approximation-Aware Bayesian Optimization and Gadd et al. Deep GPs fix this at high training cost. The matched-cost sparse-GP ablation measures whether the over-smoothing shows up as higher one-step error near contact events.

Why not just make MOPO more pessimistic? A larger $\lambda$ cannot help where disagreement is near zero, because it only rescales a signal that is absent in collapsed regions. The $\lambda$ sweep for MOPO shows whether its best setting reaches our returns.

Proposed Experiments

The decisive diagnostic comes first and needs no policy learning. We train models on each D4RL medium dataset and evaluate on transitions from the random and expert datasets of the same environment. Those transitions have true next states but lie off the training distribution. For each uncertainty signal we compute the rank correlation with true one-step error. We also compute the fraction of high-error transitions (top decile) that the signal assigns to its bottom half, which directly counts "collapse" cases. We break the results down by whether a foot contact occurs. The proposal lives or dies here. If GP variance does not track error better than ensemble standard deviation, we stop.

Second, we measure one-step and five-step rollout error with and without the residual mean, on in-distribution held-out transitions.

Third, we evaluate policies on the nine locomotion datasets against the baselines above, reporting normalized score, seed variance, and wall-clock overhead per rollout batch.

Fourth, a stress test runs the same pipeline on Ant datasets. Ant has higher state dimension, so this tests whether the bottleneck embedding keeps nearest-neighbor retrieval meaningful where the raw-input ablation should fail.