A robot that learns on real hardware has to forget. Its motors wear, its payload changes, and its sensors drift, so measurements from an hour ago describe a slightly different machine. In ordinary online learning, the standard fix is the periodic restart: throw away old data at fixed intervals so the model tracks the present. Safe exploration methods work differently. They only try actions that a probabilistic model has certified as safe, and they build that certificate up from accumulated data. If such a method throws its data away, its uncertainty jumps back to the prior. The certified safe region then disappears and the robot either freezes or has to fall back on unjustified trust. So forgetting and certified safety currently pull against each other.
This note proposes a restart that drops old data from the model's estimate but keeps a shrinking, still-valid safety guarantee across the reset. The guarantee shrinks by a known margin for each step that passes, so the robot is never left without a certificate and never relies on a stale one. Sensor drift is handled the same way, with one addition: each restart is tied to a return to a known-safe "home" configuration where the sensor's offset can be re-measured. Sensor drift is the part of the sim-to-real gap that clean simulators never show. In this design it becomes one more bounded source of change instead of an unmodeled bias.
Core Claim
Safe Bayesian optimization models an unknown safety function $g(x)$ over controller parameters or states $x$ with a Gaussian process, a probabilistic regression model that returns a mean prediction $\mu(x)$ and an uncertainty $\sigma(x)$. It only evaluates points whose lower confidence bound $\mu(x)-\beta\sigma(x)$ is nonnegative. Safe Exploration for Optimization with Gaussian Processes introduced this rule, and Kim et al. formalized it. We suppose $g_t$ drifts by at most $\epsilon$ per step in sup-norm. We also suppose the safety signal is read through a sensor with additive bias $b_t$ that drifts by at most $\delta$ per step. The proposal keeps two bounds. The first is a fresh bound computed only from the current epoch's data. The second is a carried bound inherited from the previous epoch. With $\tau$ the end of the previous epoch and $t$ the current step, the carried bound is
$$ \ell^{\text{carry}}_t(x) = \mu_{\text{old}}(x) - \beta\sigma_{\text{old}}(x) - (t-\tau)(\epsilon+\delta) - H(\epsilon+\delta), $$
where $\mu_{\text{old}},\sigma_{\text{old}}$ are the old epoch's posterior and $H$ is the epoch length. The final term covers drift inside the old epoch itself. The certified safe set is the set of points where the larger of the fresh and carried bounds is nonnegative. This bound is valid with the same probability as the original confidence bound. Old data therefore never biases the new estimate. It only contributes a pessimistic floor that decays at a known rate. If fresh data do not re-certify a region, the safe set retreats toward the home set. It neither collapses at the reset nor stays inflated.
The restart interval now plays a concrete role. It trades the statistical width $\beta\sigma$, which shrinks roughly as $1/\sqrt{H}$ for repeated nearby samples, against the drift margin $H(\epsilon+\delta)$. Minimizing their sum gives $H^\star \propto (\epsilon+\delta)^{-2/3}$. This is the same scaling that appears in the restart analyses of Gur et al.. Here it governs the size of the safe set, not regret. That gives a falsifiable prediction.
Relation to Prior Work
Two papers motivate the proposal and sit closest to it. R-GP-UCB restarts a Gaussian process in blocks and proves dynamic regret bounds for time-varying optimization, and Near-Optimal Algorithm for Non-Stationary Kernelized Bandits sharpens this line of work. Both optimize reward with no constraint, so the variance jump after a reset costs them only some regret. For a safe explorer, the same jump erases the safe set. ST-SafeMDP does handle time-varying safety. It uses a spatio-temporal kernel and a worst-case lower bound on the future safe set. However, it never discards data, so computation grows with the full history. It also has no way to separate a drifting sensor offset from a real change in safety. We keep its returnability requirement, the idea that the agent must always be able to get back to known-safe states. We replace its continuous temporal smoothing with explicit restarts and recalibration.
A second group constrains the design. PROPD-PPO combines periodic restarts with primal-dual constrained RL, which balances reward against a cost penalty whose weight is learned. It accepts cumulative violations growing as $\tilde{O}(M^{2/3})$ over $M$ episodes, which shows that naively porting restarts into safe RL gives up zero-violation exploration. Sim-to-Lab-to-Real certifies a shielded policy before deployment, but its PAC-Bayes bounds assume the lab and deployment environments are drawn i.i.d. from the same distribution. Post-deployment sensor drift breaks that assumption. Provably Safe Sim-to-Real Transfer certifies the regions where the simulator is wrong, but only in tabular settings and only for discrete, separable dynamics shifts. Neither handles a signal that degrades continuously over time. CASRL and ActSafe scale safe learning further, CASRL through context inference and ActSafe through pessimistic ensembles. Both assume a static measurement channel and give heuristic rather than confidence-bound guarantees. The proposal deliberately stays in the low-dimensional regime where Gaussian-process guarantees are real. Examples include controller gains, as in Berkenkamp et al.; barrier-function parameters, as in Optimal Parameter Adaptation for Safety-Critical Control via Safe Barrier Bayesian Optimization; and charging protocols, as in Hirt et al..
Research Program
The first deliverable is the algorithm and its safety proof. The algorithm runs SafeOpt-style expansion and optimization inside each epoch. At each boundary it does three things. It returns to the home set and takes a calibration reading against a reference whose true value is known, such as a fixed distance target for a range sensor or a seed parameter with a known margin. It subtracts the estimated offset from subsequent measurements. It then starts a fresh Gaussian process while storing the previous posterior's lower bound on a fixed set of points.
The proof shows that the combined safe set contains only truly safe points with probability $1-\delta_{\text{conf}}$ over the whole horizon. The argument is a union bound across epochs, with the calibration error added to the drift margin. The proof also states exactly when the home set stays safe. The home set must have margin larger than the drift accumulated between calibrations. This is the "safe seed" assumption made explicit, and it is not hidden.
The second deliverable is a drift-triggered variant. Instead of a fixed $H$, it restarts when fresh observations fall outside the carried bound by more than the noise level. This addresses the known brittleness of fixed epoch schedules. A wrong flag here costs only an early restart, never safety.
The third deliverable is an analysis of misspecification. We will measure how violations grow as $\epsilon$ or $\delta$ are underestimated, since kernel and smoothness misspecification is the known weak point of this whole family.
Baselines
The naive reference point is SafeOpt that never forgets. It keeps all data and treats the safety function as static. Under drift we expect it to certify points that have since become unsafe, and to violate constraints exactly when wear accumulates. Its sister baseline is SafeOpt with hard periodic restarts in the style of R-GP-UCB, which should show the predicted safe-set collapse after each reset. Both run with the same kernel, the same $\beta$, and the same evaluation budget as our method. Kernel hyperparameters are fit once on the same pilot data for all methods.
The strongest existing competitor is ST-SafeMDP. It is the only method in the file that certifies time-varying safety. On its lunar benchmark it reports zero safety violations and a precision of 1.00, meaning no unsafe states were classified as safe, while baselines that ignored temporal variation failed up to 36% of the time. We will run it on the same benchmark, with its temporal lengthscale tuned on the same pilot budget. To make compute comparable we will also run it with a sparse approximation whose inducing-point count matches our per-epoch data size. Beating it means one of two outcomes. The first is equal zero violations with a larger certified safe set or lower computation per step on long horizons. The second is zero violations where it violates, once sensor bias drift is injected, which is a regime its kernel cannot identify. No runtime figure for ST-SafeMDP is reported in the material, so the compute comparison will be measured fresh.
For the constrained-RL perspective we include PROPD-PPO on the tabular and gridworld variants. There we count total violations. Beating it means zero violations against its nonzero count at comparable reward. Two ablations isolate the key ingredients. Restart-plus-carried-bound without recalibration isolates the role of the home calibration under sensor drift. Recalibration without restarts isolates the role of discarding data under physical drift. A sliding-window variant carrying the same drift margin tests whether block restarts, rather than the margin itself, matter.
Why not just…?
Why not just use a time-aware kernel, as ST-SafeMDP does? This is the most serious alternative. A temporal kernel already inflates uncertainty smoothly as data age, which is close in spirit to our decaying margin. We see two concrete differences. First, the temporal kernel never drops data, so its cost grows with the cubic GP scaling that the file flags as limiting long deployments. Second, it models a sensor offset as a change in the safety function itself, so a slowly drifting bias gets absorbed into the posterior mean and goes uncorrected. If the matched-compute sparse ST-SafeMDP matches our safe-set size and violation count on the bias-drift benchmarks, the restart contribution reduces to a cheaper approximation. We would then report the calibration mechanism as the real contribution.
Why not just keep all data and inflate the confidence multiplier $\beta$? A larger $\beta$ widens bounds uniformly but does not remove the bias of a stale mean. Under directional wear, the mean stays confidently wrong while the true constraint moves. The never-forget baseline with $\beta$ tuned for zero violations on a held-out drift profile will show what this costs. We expect it to need a $\beta$ so large that the safe set barely grows.
Why not just filter the sensor with a Kalman filter or moving average? Filtering removes high-frequency noise but cannot observe a slowly moving zero-mean offset without a reference. It also produces no epistemic uncertainty from which a confidence bound can be built, a point Kim et al. makes precise. The recalibration-free ablation, run with filtered inputs, directly tests whether filtering alone keeps violations at zero under drift.
Why not just train a robust policy with domain randomization and sensor-noise injection? Randomization helps on average but certifies nothing. Provably Safe Sim-to-Real Transfer reports that unconstrained simulator-based transfer was unsafe in 68.5% of learning iterations. Randomization also only covers the perturbations someone thought to sample. We include a randomized controller as an external reference on the Safety-Gymnasium variant.
Proposed Experiments
The first testbed is synthetic. Safety functions are sampled from a known kernel and drift at controlled rates $\epsilon$. Additive sensor offsets drift at rate $\delta$, sometimes with abrupt steps. Because the true safe set is known, we can measure certified safe-set size as a fraction of the true safe set, violation counts, and the $H^\star \propto (\epsilon+\delta)^{-2/3}$ prediction directly.
The second testbed reuses the lunar south-pole terrain from ST-SafeMDP. We add a drifting bias to the illumination sensor. This is the setting where its reported zero-violation result should break and ours should hold.
The third testbed tunes two to four controller parameters on drifting MuJoCo systems from RRLS, introduced in RRLS : Robust Reinforcement Learning Suite. We use the inverted pendulum and the hopper, with mass and friction scheduled to change over time and a joint-limit constraint read through a drifting encoder. No reported result was found on Lacuna for RRLS, so all its numbers will be produced by this study.
The fourth testbed tunes a navigation controller's speed and clearance gains on Safety-Gymnasium, with a range bias added to its corrected lidar sensor. For context, that paper reports a 98% violation reduction for PPO-Lagrangian over PPO on velocity tasks, at a 45% reward cost. The 5x5 gridworld of Provably Safe Sim-to-Real Transfer serves as the tabular arena for the PROPD-PPO comparison. That paper reports 0% unsafe iterations under a static windy-cell mismatch, and we make the wind strengthen over time.
Four outcomes are decisive. The first is that hard-restart SafeOpt loses most of its certified set at every boundary while our method does not. The second is that never-forget SafeOpt violates constraints under drift while our method records zero. The third is that, with sensor bias drift present, the no-recalibration ablation and the temporal-kernel baseline violate while the full method does not. The fourth is that the empirically best fixed epoch length follows the predicted $-2/3$ power law across drift rates. A failure of the fourth check would mean the bias-variance picture is wrong for safe sets, even if the safety proof holds. The misspecification sweep will report honestly how quickly violations appear once the assumed drift rate is set below the true one. That measurement is the main practical limitation a user of the method would face.
References
- Safe Exploration for Optimization with Gaussian Processes
- Safe Learning and Optimization Techniques: Towards a Survey of the State of the Art
- Stochastic Multi-Armed-Bandit Problem with Non-stationary Rewards
- No-Regret Algorithms for Time-Varying Bayesian Optimization
- Near-Optimal Algorithm for Non-Stationary Kernelized Bandits
- Safe Exploration in Markov Decision Processes with Time-Variant Safety using Spatio-Temporal Gaussian Process
- Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints
- Sim-to-Lab-to-Real: Safe reinforcement learning with shielding and generalization guarantees
- Provably Safe Sim-to-Real Transfer
- Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
- ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning
- Safe Controller Optimization for Quadrotors with Gaussian Processes
- Optimal Parameter Adaptation for Safety-Critical Control via Safe Barrier Bayesian Optimization
- Safe Learning-Based Optimization of Model Predictive Control: Application to Battery Fast-Charging
- Rrls: Robust reinforcement learning suite
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark