Safe reinforcement learning trains robots to pursue a task while keeping a separate "cost", such as collisions or entering forbidden zones, under a budget. The benchmarks used to test these methods assume the robot always knows exactly where it is. Real robots working without external tracking hardware (motion-capture cameras, surveyed radio beacons, GPS) instead estimate their position from onboard cameras and motion sensors. Those estimates drift over time. When robots work in teams, they can correct each other by measuring the distances between them with short-range radio.
The errors in these estimates are not random jitter. They grow when the robot moves aggressively, because camera features blur. They grow near obstacles, because walls block the line of sight between robots and corrupt the distance measurements. So the errors are largest exactly when and where a mistake is most dangerous. A policy that is safe with true positions can therefore be unsafe with estimated ones, and the gap depends on the policy's own behavior.
This note proposes two things. The first is a benchmark extension of Safety-Gymnasium in which a realistic team localization estimator sits inside the control loop. The second is a safety-margin method that sizes its caution from signals the estimator can compute about its own health. The method is calibrated in simulation and tested under shifts in sensor quality, where both unmodified safe-RL methods and methods that learn caution implicitly are expected to fail. The central experiment is designed so that it can also show the method is unnecessary. If it does, we will report that.
Core Claim
In infrastructure-free teams, position error that matters for safety is endogenous: the policy's speed and its placement relative to obstacles and teammates determine how badly localization degrades. We claim three things follow. First, safe RL algorithms scored with ground-truth state overstate their real-world safety by an amount that current benchmarks cannot measure. Second, a safety margin conditioned on the estimator's self-consistency signals transfers across sensor-quality regimes better than two alternatives: a fixed margin, and a policy that has only implicitly learned the simulated error distribution. These signals are the disagreement between measured inter-robot ranges and estimated positions, the number of teammates in line of sight, and camera-tracking quality. Third, because a policy can influence those signals, it can learn to stay localizable, for example by slowing near walls or keeping a teammate in view, rather than simply freezing.
Relation to Prior Work
The closest work is SPiDR. SPiDR adds to the safety cost a penalty proportional to the disagreement among an ensemble of randomized physics simulators. This gives zero-shot constraint satisfaction on a Unitree Go1 and a small RaceCar, where standard domain randomization produced 26.1x and 6.2x more violations respectively. We reuse its additive-pessimism structure. SPiDR, however, randomizes dynamics only, and its hardware relies on existing tracking. Its penalty measures model uncertainty in simulation, not state uncertainty at deployment. We replace the dynamics-variance term with a bound on position error predicted from runtime estimator signals.
Provably Safe Sim-to-Real Transfer gives zero-violation guarantees by learning where simulator and reality differ. It is tabular, assumes the discrete cell is observed exactly, and was evaluated on a 5x5 gridworld. It motivates the goal but constrains our ambitions: we do not claim guarantees of that kind in continuous state.
Safe-RL algorithms supply the optimizers we build on. The Lagrangian and projection methods in SafePO, bundled with Safety-Gymnasium, are one source. So is FOCOPS, a first-order constrained policy update. So are Backward Value Functions, which turn a whole-episode cost budget into per-state constraints so safety holds during training and not only at convergence. All of them assume exact state. The BVF authors note that conservative per-state bounds cost performance. Our method is one more source of conservatism of exactly that kind, so this is a live risk we measure.
The localization side supplies the estimator and the error structure. Decentralized and Fully Onboard range-aided MAV localization fuses onboard visual-inertial odometry (position tracking from a camera plus accelerometers and gyroscopes) with inter-drone ultra-wideband ranging in a decentralized factor graph. It reaches about 0.1 m positioning error, but explicitly does not address obstacle avoidance. SaWa-ML shows that one robot's odometry scale drift during aggressive motion can warp a whole team's estimate. Its fix is to use range measurements as a drift-free geometric skeleton that down-weights inconsistent robots, cutting absolute trajectory error from 6.14–6.97 m to 3.83 m in one outdoor sequence and from 5.23–7.91 m to 2.69 m in another. Both are passive estimators with no safety or learning component. SaWa-ML's consistency weights are precisely the kind of self-health signal we propose to feed into the safety margin.
Control-Barrier-Aided Teleoperation with VI-SLAM shows a single-drone safety filter on estimated state that avoided all of 83 would-be collisions in simulation. It trusts the SLAM pose, however, and has no multi-robot ranging. METRICMM down-weights corrupted sensor streams in unconstrained RL. It motivates using consistency signals rather than Gaussian noise models, but it has no cost budget, and it assumes symmetric distances and deterministic transitions that do not fit here.
Research Program
The benchmark. We extend Safety-Gymnasium's navigation tasks (Goal and Button, with Point and Car agents) to teams of three to five agents, using the MJX parallel port introduced by SPiDR for throughput.
Each agent receives only its estimated pose and the estimates of its peers. These come from a decentralized range-aided estimator modeled on Goudar and Schoellig. The estimator's inputs are:
- a simulated visual-inertial odometry stream whose drift rate grows with angular velocity and with a texture parameter per region;
- inter-agent ranges that pick up a positive bias and heavy-tailed errors whenever the straight segment between two agents intersects an obstacle, which is the standard non-line-of-sight failure.
Costs are always scored on true state. Hazard zones are expressed in the team's shared frame, as if mapped earlier by teammates, and minimum inter-agent separation is also enforced. This choice matters. Drift common to all agents partly cancels against a map built in the same drifting frame. What drives violations is differential error between agents, and that is what relative ranging can observe.
The benchmark reports reward and cost under ground-truth state and under estimated state. The difference between the two is the benchmark's headline number.
The method. At each step the agent computes a vector of estimator signals, $z_t$:
- range residuals, i.e. the mismatch between measured distances and distances implied by current estimates;
- the count of line-of-sight neighbors;
- tracked-feature count;
- speed.
A small network $g$ maps $z_t$ to a predicted upper bound on position error. That bound is calibrated in simulation by split conformal prediction over many randomized error regimes. The shaped cost used during training and deployment is
$$\tilde c(\hat s_t, a_t) = \mathbb{1}\left[\, d(\hat p_t, \mathcal{H}) < r + q_{1-\alpha}\big(g(z_t)\big) \right],$$
where the terms are defined as follows:
- $\hat p_t$ is the estimated position;
- $\mathcal{H}$ is the set of hazards and teammates;
- $d$ is distance;
- $r$ is the nominal clearance;
- $q_{1-\alpha}$ is the conformal quantile giving $1-\alpha$ coverage of true error.
This cost plugs into PPO-Lagrangian or FOCOPS unchanged. Because $g(z_t)$ depends on the agent's own actions, the constrained optimizer is rewarded for keeping itself localizable. We state the limit of the coverage guarantee plainly. It holds only under exchangeability with calibration data, which a changing policy breaks, so we measure empirical coverage rather than assume it.
A small hardware check. Ground robots carrying UWB radios and stereo cameras will navigate a cluttered room. Motion capture is used only to score violations, never in the loop. This inverts the usual role of lab infrastructure: it becomes the referee, not a crutch.
Baselines
Oracle-state reference. PPO-Lagrangian from SafePO, trained and tested on true state. This is the "do nothing about localization" reference and represents what current benchmarks report. On Safety-Gymnasium velocity tasks, the benchmark paper reports that PPO-Lag gave a 98% reduction in violations over PPO with a 45% reward drop. Running it with estimated state at test time shows the size of the hidden gap.
Observation-noise domain randomization. The same learner, trained with independent Gaussian noise added to positions.
Fixed-margin and covariance-margin controllers. A constant inflated clearance tuned per task, and a clearance scaled by the factor graph's reported covariance.
SPiDR. Its dynamics-ensemble penalty, run with the estimator in the loop. SPiDR reports 100% real-world constraint satisfaction on its platforms, and it is the strongest published safe-transfer method in the file. No reported result was found on Lacuna for any safe-RL method trained or evaluated with an infrastructure-free estimator in the loop, so all numbers in our setting will be new.
Estimator-in-the-loop Lagrangian. PPO-Lagrangian trained directly on estimated observations with cost on true state. This is the strongest competitor. It can learn caution implicitly, including action-dependent caution, with no explicit margin.
Ablation. The central ablation replaces $g(z_t)$ with a margin that depends only on speed. This separates the value of estimator-health signals from simple slow-down behavior.
Fairness. All methods share the same environments, the same estimator, 10 seeds, the same environment-step budget, and the same hyperparameter-search budget. The search budget includes tuning SPiDR's $\lambda$ and the fixed margin.
What counts as beating a baseline. Our method must achieve lower true-state cost at equal or higher reward than every baseline in the shifted regimes described below. It must also match the estimator-in-the-loop Lagrangian within the training regime.
Why not just…?
Why not just train the safe-RL agent with the estimator in the loop and let it learn caution? This is the real threat. Within the training error distribution it may match or beat us, because it sees the same signals implicitly. Our claim concerns shift. What the implicit policy learns is calibrated to simulated error statistics, while an explicit bound on self-consistency signals should degrade more gracefully when real sensors are worse than simulated. The shift experiment settles this directly. If the implicit learner matches us under shift, the method adds nothing beyond the benchmark, and we will say so.
Why not just inflate obstacles by a fixed margin? A margin large enough for the worst case, near walls with blocked line of sight, makes the policy overly cautious everywhere else. This is the "safe but useless" failure that Quanyi Li et al. document for hand-tuned penalties. The reward-versus-cost frontier comparison will show whether a state-dependent margin actually dominates.
Why not just use the factor graph's covariance? Estimator covariances describe Gaussian noise. SaWa-ML shows that standard fusion confidently absorbs one robot's scale drift into the whole team, which is the kind of error covariance misses. METRICMM makes the same point for non-Gaussian sensor failures. We will report empirical coverage of covariance-based bounds against conformal ones.
Why not just use domain randomization? SPiDR's hardware results, 26.1x more violations on the Go1 and 6.2x on the RaceCar with standard randomization, already show that making a policy robust is not the same as respecting a constraint. Independent noise also cannot represent error that depends on where the robot goes and how fast.
Decisive Checks
The first check quantifies the hidden gap. For every SafePO algorithm and FOCOPS, we measure the ratio of true-state cost with estimated observations to cost with oracle observations. A ratio near one would undercut the benchmark's motivation.
The second check tests the endogeneity claim. We fit per-agent position error on distance-to-obstacle, line-of-sight count and speed in collected rollouts. If these variables explain little of the variance, then independent-noise randomization is adequate and the method's premise fails.
The third check is the shift test. We train at one non-line-of-sight bias level and odometry drift rate, then evaluate at two and four times each. We compare violation rate and reward across all baselines, and track empirical conformal coverage.
The fourth check is behavioral. We examine whether learned policies change line-of-sight maintenance and speed near hazards relative to the speed-only ablation.
The fifth check is the hardware test. We compare motion-capture-scored violation counts between our method and the estimator-in-the-loop Lagrangian over matched runs.
References
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Provably Safe Sim-to-Real Transfer
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- First Order Constrained Optimization in Policy Space
- Constrained Markov Decision Processes via Backward Value Functions
- Decentralized and Fully Onboard: Range-Aided Cooperative Localization and Navigation on Micro Aerial Vehicles
- SaWa-ML: Structure-Aware Pose Correction and Weight Adaptation-Based Robust Multi-Robot Localization
- Control-Barrier-Aided Teleoperation with Visual-Inertial SLAM for Safe MAV Navigation in Complex Environments
- Geometry of Uncertainty: Learning Metric Spaces for Multimodal State Estimation in RL
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning