When a human drives a robot over a slow network, what they see is always somewhat out of date, and the commands they send arrive late. A learned safety filter is a natural fix: a model with calibrated uncertainty that blocks any command likely to lead somewhere dangerous. The obvious way to add one is to run it on the operator's side and look further ahead to cover the delay. That design pays for the delay twice. Uncertainty compounds over the longer look-ahead, so the filter becomes very conservative, and under variable delay it has to plan for the worst case. We propose splitting the job. The safety guarantee is enforced on the robot, which sees its own state with no delay. The operator station runs a probabilistic forecast of what that remote filter is about to do, and turns that forecast into haptic resistance and visual cues before the delayed feedback arrives. With this split, delay no longer weakens the guarantee. It only affects how good the preview feels, and that is a far cheaper place to absorb it.
This matters most when the operator is also the explorer. In remote inspection, rover driving on delayed links, or telesurgery, the operator pushes into regions the safety model has never seen, and the model's certified-safe region grows from that data. If the operator cannot anticipate where the robot-side filter will intervene, they keep driving into the boundary. Each rejection shows up a full round trip later and causes an overcorrection. The safe region then grows slowly, because boundary visits turn into oscillating fights with the filter rather than informative probes. Our claim is that forecasting the filter's interventions under delay improves both operator performance and the rate of safe exploration, while keeping the same high-probability safety guarantee as a delay-free filter.
Core Claim
The work builds on recovery shielding with Gaussian processes. A Gaussian process (GP) is a Bayesian regression model that returns a prediction together with an uncertainty band that widens away from observed data. In Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models, a GP learns the dynamics online. The method propagates confidence ellipsoids over an $N$-step look-ahead using analytic moment matching, and it overrides a learning agent whenever a pre-computed backup controller could no longer return the system to a known invariant set. It assumes observation and actuation happen in lockstep.
Our proposal keeps that filter unchanged on the robot. There it checks each command against fresh state as the command arrives, so its guarantee horizon stays at $N$ rather than growing to $N+2\tau$.
The new component is an operator-side forecaster. Let $\tau_d$ be the downlink delay and $\tau_u$ the uplink delay. The operator last saw state $x_{t-\tau_d}$ and knows every command it has sent that is still in flight. For a candidate command $u$, the forecaster estimates the probability that the robot-side filter will reject $u$ when it arrives:
$$p_{\text{ov}}(u) = \Pr\big[\, \text{Shield}(x_{t+\tau_u}, u) = \text{reject} \;\big|\; x_{t-\tau_d},\, u_{t-\tau_d:t-1},\, \mathcal{D}_{t-\tau_d} \big].$$
Here $x_{t+\tau_u}$ is the robot state when $u$ arrives, $u_{t-\tau_d:t-1}$ are the in-flight commands, and $\mathcal{D}_{t-\tau_d}$ is the most recent copy of the GP training data the operator has received. The forecaster computes this by pushing the delayed state forward through the same moment-matched GP for $\tau_d+\tau_u$ steps, and it models the chance that earlier in-flight commands were themselves overridden. The output is rendered as resistance proportional to $p_{\text{ov}}$.
Errors in this forecast cost usability, never safety. That asymmetry is what makes the design attractive: the compounding variance that cripples an operator-side filter only makes the preview blurrier.
A second, smaller claim concerns exploration. The robot-side safe set expands in the SafeMDP style, and those updates reach the operator late. The forecaster can also display the current "expanders", meaning boundary states whose visit would shrink uncertainty enough to grow the safe set. Doing so turns the human into a directed explorer instead of a random one.
Positioning Against Prior Work
Three teleoperation papers motivate this proposal and supply parts of the design.
- Bimanual telemanipulation with force and haptic feedback through an anthropomorphic avatar system runs a local predictive model that gives haptic resistance before joint and velocity limits are reached. It scored 99 of 100 points at the ANA Avatar XPRIZE semifinals at about 44 ms latency. Its limits are deterministic and known in advance. We keep the idea of resisting locally and early, but we replace fixed limits with a forecast of a learned, uncertain, growing constraint, at much larger delays.
- Rosenstein et al. project the robot's future path on predictor displays. We project the future decisions of a safety filter instead of a trajectory.
- Optimal Motion Scaling for Delayed Telesurgery fits a Bayesian regression that chooses a per-user, per-delay motion scale. It showed that scales below 0.5 reduce error under delay. It is a static mapping with no state-space safety guarantee, and it is our main simple competitor.
- Enhancing the Force Transparency of the Energy-Reflection-Based Time-Domain Passivity Approach reduces force attenuation at 400 ms from 50.4% to 17.4%. It addresses stability of the force loop, not state constraints. We treat it as complementary and do not compete with it.
On the safety side, three papers supply the method and set its constraints.
- Safe Exploration in Finite Markov Decision Processes with Gaussian Processes supplies the expander and returnability logic, the idea that a state is only admitted if the robot can safely get back from it. On Mars HiRISE terrain it reported 80.28% coverage of safely reachable area versus 30.44% without expanders. It assumes an explorer that acts instantly, and a delayed human is not one.
- The recovery-shielding paper above supplies the robot-side filter. It also brings its known weakness: dependence on a linear backup controller that becomes conservative on high-dimensional systems such as Hopper. We inherit that weakness rather than solve it.
- Breaking Safety Paradox with Feasible Dual Policy Iteration shows that safety estimators starve for data near the boundary once a system becomes safe. This is a warning for our design: a preview that is too cautious could keep the operator away from exactly the data the GP needs. We therefore measure safe-set growth directly.
Learning Robot Safety from Sparse Human Feedback using Conformal Prediction supplies a tool for calibrating the forecaster. Conformal prediction is a distribution-free way to turn prediction errors into coverage-guaranteed bands, and its codebase is available at StanfordMSL/conformal-safety-learning. Finally, SAFe-Copilot arbitrates human and autonomous plans with a vision-language model. Its authors flag inference latency as a bottleneck, and it gives no formal bound. Our arbitration is a closed-form GP computation with a guarantee enforced on the robot.
Research Program
The system has three components.
Robot-side filter. This is the recovery shield with a sparse GP dynamics model, which addresses the cubic cost of exact GPs. It is extended with a hold behavior for packet dropouts: if no fresh command arrives, the backup controller acts. This preserves the original guarantee under any delay distribution, because the filter never uses stale state.
Operator-side forecaster. This component receives delayed state and delayed GP updates. It maintains a moment-matched belief over the current remote state and over the current version of the safe set. It computes $p_{\text{ov}}$ for the operator's current command and for nearby commands, and renders the gradient of $p_{\text{ov}}$ as haptic resistance. Under jitter it propagates over the empirical delay distribution rather than a worst-case delay. We then calibrate $p_{\text{ov}}$ with conformal prediction against realized overrides.
Exploration cue. The operator display marks SafeMDP-style expanders inside the forecast safe region.
On the analysis side, we will state and check two properties. The first is that the safety probability of the split system equals that of the undelayed robot-side filter. This follows because the filter's inputs are unaffected by the link, and it is the claim we want tested empirically, not just asserted. The second is that the forecaster's calibration error grows with $\tau_d+\tau_u$ at a rate set by GP variance compounding. That growth tells us at what delay the preview stops being useful, and we want to measure that crossover rather than assume it away.
Baselines
Naive reference. Direct delayed teleoperation with no filter and no scaling. This sets the violation rate and oscillation level the problem starts from.
Speed-limit reference. A fixed global speed cap tuned to zero violations on held-out trials. It is the most common industrial answer.
Motion scaling. Per-user Bayesian motion scaling as in Optimal Motion Scaling for Delayed Telesurgery, fit on the same calibration budget our forecaster gets. It is the strongest cheap competitor because it already adapts to delay and to the individual operator. That paper reports only significance results (error reduction at scales below 0.5, $p<0.05$). No absolute peg-transfer error is reported on Lacuna, so we will measure it ourselves. Beating it means fewer violations at equal or better task time, and faster safe-set growth.
Operator-side filter. The recovery shield from Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models, placed at the operator station with its look-ahead extended by the round-trip delay (worst-case under jitter). This is the "obvious" intersection design and the strongest method-level competitor. Undelayed, the source reports 1.000 safety probability across all ten tasks and the highest mean reward on 7 of 10 tasks versus CPO and PPO-Lag. Delayed results are not reported, and we will produce them. Beating it means matching its safety while achieving higher task return or coverage. We expect it to over-restrict as delay grows.
Robot-side filter alone. The same shield on the robot with no operator preview. This ablation isolates the forecaster: any gain of the full system over it is attributable to the preview.
Mean-only preview. A preview that renders the deterministic predicted override, as in Lenz-style limit avoidance, without uncertainty. This ablation isolates the probabilistic ingredient.
Fairness conditions. All learned components share the same GP kernel family, sparse inducing-point budget, and hyperparameter tuning budget. All systems use the same delay traces, and human participants see conditions in counterbalanced order.
Why not just…?
Why not just extend the operator-side filter's look-ahead by the delay? This is the design we most expect to lose. Pushing moment-matched ellipsoids over $N+2\tau$ steps compounds variance, and jitter forces a worst-case $\tau$. The recovery-shielding paper already notes conservatism on Hopper at zero delay. The operator-side-filter baseline tests this directly by sweeping delay. If its return and coverage stay close to the split system's up to 400 ms, the split is unnecessary, and we would report that.
Why not just put the filter on the robot and do nothing else? This is a real threat, because it already gets the safety guarantee. What it lacks is any signal to the operator before an override has happened and travelled back. Teleoperation work (Lee et al.) predicts that late, surprising corrections cause oscillation. The robot-side-alone ablation settles the question. If override counts, oscillation energy, task time, and safe-set growth do not improve with the preview, the forecaster is not worth its complexity, and the paper's conclusion becomes "place the filter remotely".
Why not just slow everything down with motion scaling or a speed cap? Slowing reduces overshoot, but it does not know where the constraints are and gives up throughput everywhere. The GPA-Teleoperation comparison illustrates the cost: a speed-clamped commercial drone needed 31.2 s against 7.5 s for the perception-aware system. SafeMDP's no-expander variant stayed safe but covered only 30.44% of reachable terrain. Our comparison against Bayesian motion scaling at matched calibration budget decides whether state-dependent resistance beats uniform slowing.
Why not just add a large collision penalty to a learned assistant? Quanyi Li et al. show this oscillates between accepting violations and freezing. Safety Gymnasium reports that even a constrained method, PPO-Lag, keeps its 98% violation reduction only at a 45% reward drop, with no per-episode guarantee. For these reasons we do not run a penalty-only baseline beyond the naive reference.
Proposed Experiments
We run three testbeds of increasing realism.
Mars rover driving. We use the HiRISE Mars elevation map from SafeMDP (120 m by 70 m, failure above 30°). A simulated rover is driven over links with constant delays of 0, 1, 2, and 5 seconds and with jittered delays. The explorer is either a scripted delayed operator model or a human. The primary metric is the percentage of truly safe reachable area certified after a fixed operating time, directly comparable to SafeMDP's 80.28% delay-free figure. Violations must remain zero.
Continuous control with delay. We take the recovery-shielding suite (obstacle 1–4, road_2d, cartpole) and Safety-Gymnasium Point-Goal and Car-Goal with injected delay of 100–400 ms. Metrics are empirical safety probability, return, override rate, oscillation power in the command signal, and calibration of $p_{\text{ov}}$ against realized overrides.
Human study. A haptic manipulation task follows the peg-transfer structure used on the da Vinci Research Kit in the motion-scaling study, run in simulation with injected delay. Metrics are task time, positioning error, overshoot force, override count, and workload.
Decisive checks. Three results decide the proposal. First, the split system must keep the delay-free safety probability at every delay while the operator-side filter loses return or coverage as delay grows. Second, the full system must beat the robot-side-only ablation on safe-set growth and oscillation; this is the result that justifies the forecaster. Third, the probabilistic preview must beat the mean-only preview once delay is large enough that forecast uncertainty is substantial. We will also report the delay at which forecaster calibration collapses, because beyond that point the honest recommendation is the plain robot-side filter.
References
- Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models
- Bimanual Telemanipulation with Force and Haptic Feedback through an Anthropomorphic Avatar System
- Remote Supervisory Control of a Humanoid Robot
- Optimal Motion Scaling for Delayed Telesurgery
- Enhancing the Force Transparency of the Energy-Reflection-Based Time-Domain Passivity Approach
- Safe Exploration in Finite Markov Decision Processes with Gaussian Processes
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Learning Robot Safety From Sparse Human Feedback Using Conformal Prediction
- SAFe-Copilot: Unified Shared Autonomy Framework
- Visual-Inertial Telepresence for Aerial Manipulation
- GPA-Teleoperation: Gaze Enhanced Perception-Aware Safe Assistive Aerial Teleoperation
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark