Pessimism About Other Agents: Safety Critics for Sim-to-Real Transfer in Moving Crowds

Robots trained in simulation usually learn to be safe around walls, furniture, and parked cars, because that is what their simulators contain. The best current methods for keeping a simulation-trained policy safe on real hardware make one of two choices. Some penalize the robot wherever randomized physics simulators disagree. Others learn a "safety critic", a network that scores how close each state is to an unrecoverable crash. Both have been tested only in scenes where nothing else moves. In the real world, the largest and least predictable gap between simulation and reality is often other agents: pedestrians, drivers, other robots. Simulators drive these agents with simple scripted rules, and real agents do not follow those rules.

We propose to train the safety critic against an ensemble of disagreeing models of how other agents behave, and to back it up pessimistically. A state counts as dangerous if any plausible behavior model can turn it into an unavoidable collision a few seconds later. In two sentences: current safe sim-to-real methods hedge against wrong physics but not against wrong assumptions about other agents' behavior. We hedge against wrong behavior by having the robot's learned safety estimate assume the worst over a family of agent behavior models, and we test whether this transfers to unseen real traffic better than averaging over randomized behaviors or bolting on a runtime planner.

Core Claim

The claim is specific. When a safe actor-critic agent is trained in simulation with scripted moving agents and deployed against agents whose behavior differs, most constraint violations come from behavior mismatch, not physics mismatch. Physics-disagreement pessimism of the kind used by SPiDR will therefore not remove them. Plain randomization of behavior parameters will not remove them either, because averaging over behaviors trains a policy for the typical agent and not the dangerous one. A reachability-style safety critic whose Bellman backup takes a low quantile over an ensemble of behavior models should remove a large share of them, without the "freezing robot" failure that uniform safety margins produce.

The mechanism reuses the discounted safety Bellman equation from Sim-to-Lab-to-Real. That work trains a critic to predict a continuous safety margin, not a binary crash flag, and uses it to shield a task policy, but only in static indoor scenes. We change the backup so it ranges over $K$ behavior models of the surrounding agents:

$$Q_{\text{safe}}(s,a) \leftarrow (1-\gamma)\,g(s) + \gamma \min\Big\{ g(s),\; \mathrm{q}_\alpha\big[\max_{a'} Q_{\text{safe}}(s'_k,a')\big]_{k=1}^{K} \Big\}$$

Here $g(s)$ is the signed safety margin (distance to the nearest agent or obstacle minus a clearance radius, positive when safe). $s'_k$ is the next joint state when the other agents follow behavior model $k$. $\mathrm{q}_\alpha$ is the lower $\alpha$-quantile across the $K$ models. $\gamma$ is the discount. A negative $Q_{\text{safe}}$ marks what Schmerling et al. call an inevitable collision state: a point that is free now but from which no maneuver avoids impact once agents move. Their importance-sampling certificates detect such states only for precomputed trajectories among stationary pillars, and they name moving obstacles as future work.

The quantile $\alpha$ replaces SPiDR's hand-tuned penalty weight $\lambda$ with a parameter that has a direct meaning: how pessimistic to be about other agents.

Research Program

The first component is the behavior ensemble. In driving, its members are car-following and lane-change models with widely different parameters: a cautious model, an aggressive model, a non-yielding model that ignores the ego vehicle, a reactive model that brakes for it, and a model with random hesitation. In pedestrian navigation, the members are social-force-like crowds with different goal-seeking and avoidance strengths. Members are chosen to disagree on purpose, not sampled from one narrow distribution.

The second component is the actor-critic learner. It is a soft actor-critic (an off-policy actor-critic that adds an entropy bonus) with three heads: a reward critic, the behavior-pessimistic safety critic above, and a policy that is shielded by the safety critic. Shielding follows Sim-to-Lab-to-Real. To address that paper's noted problem of abrupt overrides causing chattering, we also add a soft constraint on the policy: a Lagrangian term that pushes the actor away from actions whose pessimistic safety value nears zero. The task actor then learns to stay clear of the boundary instead of relying on the backup.

The third component is analysis. We will attribute each test-time violation to either physics or behavior. We do this by replaying the failing episode with the same physics but the training behavior model, and again with the training physics but the test behavior model. This directly tests the central assumption that, in dynamic scenes, behavior mismatch dominates.

The testbeds are chosen so that "reality" means agents the policy never trained against. MetaDrive is the main one. Policies train on procedurally generated maps with scripted traffic and are tested on traffic imported from Waymo and Argoverse logs, which are real human driving that no scripted model produced. One caveat: logged agents do not react to the ego vehicle, so this tests a behavior gap with non-reactive real agents. A reactive held-out behavior model covers the other side.

The SUMO on-ramp setting from Combining Reinforcement Learning with Model Predictive Control for On-Ramp Merging provides a documented behavior shift: training on moderate traffic and testing on slow traffic. We will also add moving hazards to Safety-Gymnasium Goal tasks, because its navigation tasks contain essentially only static hazards. The final stage is a physical 1/10-scale race car, the platform SPiDR used, driving among remote-controlled cars and walking people.

Baselines

The naive reference is a Lagrangian soft actor-critic trained on a single scripted behavior model, with standard physics domain randomization. This is the setup the field would deploy by default, and it shows how large the behavior gap is.

The strongest existing safe sim-to-real method is SPiDR. It trains an ensemble of randomized physics simulators and adds their next-state disagreement to the safety cost. It reports "100% constraint satisfaction in the real world" on a 1/10-scale RaceCar and Unitree Go1, against "6.2x higher" violations for standard domain randomization on the RaceCar. Those results come from static obstacles. We will run SPiDR with its published ensemble size of at least eight physics models, with the same agent count, environment steps, and actor-critic backbone as our method. Its $\lambda$ gets the same number of tuning trials as our $\alpha$. Beating it means lower violation rates on held-out behaviors (logged Waymo/Argoverse traffic, slow SUMO traffic, real moving agents on hardware) at equal or better success rate.

The strongest dynamic-agent safety method is the runtime supervisor of the on-ramp paper. A space-time model-predictive controller predicts surrounding vehicles over a 5-second horizon and overrides a DDPG actor (an actor-critic method with a deterministic policy) when needed. It reports a "0% crash rate" in standard and out-of-distribution traffic, against a pure-RL agent that "crashed 85.1% of the time in 'slow' traffic when trained on 'moderate' traffic". That supervisor received perfect ground-truth future trajectories. For a fair comparison we give it the same learned trajectory predictor our method would use, plus a constant-velocity predictor, and charge its per-step planning time against the same onboard compute limit. Beating it means matching its crash rate with lower jerk and shorter merge times, or a lower crash rate once its predictions are no longer perfect.

A margin-inflation baseline treats every agent as a static obstacle enlarged by its velocity times a reaction time. A static-scene reachability shield, Sim-to-Lab-to-Real applied with a single behavior model, isolates the value of the ensemble.

Two ablations isolate the key ingredient. The first trains with the full behavior ensemble but averages the backup instead of taking the quantile. The second applies the quantile over physics models only. If averaging matches the quantile, pessimism is not the active ingredient.

Why not just…?

Why not just randomize the other agents' behavior parameters, as domain randomization does for physics? This is the most direct threat, and it is the averaging ablation above. SPiDR's own results suggest the answer: standard physics randomization produced 6.2x to 26.1x more real-world violations than its pessimistic counterpart. The reason is that an average-case objective drifts toward the center of the randomized distribution. We expect the same for behavior. If held-out violation rates under averaged behavior randomization come within noise of the quantile backup, the pessimism claim fails, and we will report it that way.

Why not just put an MPC safety supervisor on top of any RL policy? The on-ramp paper shows this works when predictions are perfect. Our experiments remove that assumption and add a compute limit, which is where we expect the supervisor's crash rate to rise. The supervisor also needs online optimization over the space-time tubes that make dynamic planning expensive on embedded hardware. If it survives imperfect predictions at matched compute, it remains the better engineering choice for highway merging, and our contribution narrows to settings where no good predictive model exists, such as crowds.

Why not just inflate safety margins around moving agents? Large margins prevent collisions by refusing to act. This is the "safe but useless" failure that MetaDrive's authors observed for over-penalized agents. We measure the fraction of time the robot is stationary and the task success rate alongside violations. Margin inflation succeeds only if it matches our violation rate without a drop in success. Dense merging and crowds, where gaps close faster than any fixed buffer allows, are the cases designed to separate the two.

Decisive Experiments

The first experiment is the attribution study on MetaDrive and SUMO. If fewer than half of the naive baseline's held-out violations are attributable to behavior and not physics, the motivation weakens and we redirect toward physics.

The second is the main comparison against all baselines on held-out behaviors. We report collision rate, constraint-cost rate, success rate, stationary-time fraction, and, in SUMO, jerk and merge time, with the SUMO reference numbers above as anchors. MetaDrive's paper reports roughly 80% success on unseen maps but no per-baseline safe-RL numbers. No reported result on Lacuna was found for safe RL on MetaDrive's imported real traffic, so all MetaDrive numbers will be produced here under one shared protocol.

The third is a sweep over $\alpha$ to show the tradeoff between violations and stationary time, compared against the tradeoff produced by sweeping SPiDR's $\lambda$.

The fourth is the hardware run: the 1/10-scale car among moving agents, compared with SPiDR under identical training budgets. Success here would extend the strongest existing safe-transfer result from static scenes to scenes with moving agents.