Pooled, Symmetric Simulator Correction for Safe Swarm Sim-to-Real

When a team of identical robots moves from simulation to hardware, the simulator is usually wrong in the same way for every robot. The motors share a lag, the floor has one friction value, and two robots passing close to each other push each other the same way whichever pair it is. Current safe sim-to-real methods ignore this structure. They correct the simulator either for one robot at a time or over the full joint state of the whole team, and the joint state grows exponentially with team size. We propose to correct the simulator with a single shared model of the error. That model has two parts: a per-robot term that depends only on one robot's own state and action, and a pairwise term that depends only on the relative state of two neighbors. Both parts use one set of weights for every robot. A real transition from any robot then certifies the simulator for all robots, and a close encounter between any one pair certifies it for every pair. Data is collected safely: each robot mixes exploratory actions with a known safe fallback controller, and a team-wide risk budget decides which robots are allowed to probe uncertain regions at any moment.

The opportunity is concrete. Provably Safe Sim-to-Real Transfer shows, for a single agent in a small grid, that the number of real-world samples needed scales with the size of the region where the simulator is wrong, not with the full state space. It also records zero unsafe iterations, where an unconstrained baseline was unsafe in 68.5% of iterations. Applied naively to a team, that "wrong region" lives in the joint space and the guarantee becomes useless. Sharing parameters across identical agents is normally used to pool experience for the policy. Here we use it to pool evidence about the simulator's errors, which is where real-world samples are scarce and dangerous to collect.

Core Claim

For a team of $N$ identical agents, write the real next-state of agent $i$ as

$$ s_i' = f_{\text{sim}}(s_i, a_i, s_{-i}) + g_\theta(s_i, a_i) + \sum_{j \in \mathcal{N}(i)} h_\phi(s_j - s_i, a_i, a_j) + \text{noise}. $$

Here $f_{\text{sim}}$ is the simulator's prediction, $s_{-i}$ is the state of the other agents, and $g_\theta$ is a shared per-agent correction. $h_\phi$ is a shared pairwise correction applied to each neighbor $j$ in agent $i$'s neighborhood $\mathcal{N}(i)$; it is symmetric under swapping which agent is called $i$.

The claim has three parts. First, the region that must be certified from real data lives in the single-agent space plus the pairwise relative-state space, not the joint space. Second, all $N$ agents contribute samples to the same two correction models, so the real data needed to certify a given region falls roughly by a factor of $N$. That factor follows from pooling counts: each agent's transition adds one sample to a shared confidence bound. Third, all of this can be done under a per-step team safety budget with zero hardware violations in our evaluation settings.

What is new relative to the tabular single-agent result is three things together. The mismatch region is factored by symmetry. Evidence is pooled across agents. Probing is allocated across agents under a shared risk budget. Relative to grounding methods such as Reinforced Grounded Action Transformation, which learns an action-correcting network from real transitions but collects them with no safety mechanism and for one agent, our data collection is constrained and the correction is multi-agent by construction.

The claim fails in a recognizable way if real errors are not well captured by the per-agent-plus-pairwise form. Examples are three-body airflow effects, or robots that are not actually identical because of manufacturing variation. We test both failure modes directly.

Research Program

Mismatch models with calibrated uncertainty. We replace the tabular "certified or not" bookkeeping of Provably Safe Sim-to-Real Transfer with ensembles of small networks for $g_\theta$ and $h_\phi$. A local region counts as certified when the ensemble's confidence bound on the residual falls below a margin. That paper's separable-shift assumption, under which the simulator error is either zero or clearly nonzero, does not hold for gradual continuous drift. We therefore report coverage of the confidence bounds rather than assume them. To test robustness to robots that are not quite identical, we add an optional small per-agent offset to $g_\theta$, fitted with strong shrinkage toward the shared value.

Hybrid simulator. Policy training uses a hybrid simulator, again following the tabular construction. In certified regions it uses $f_{\text{sim}}$ plus the learned corrections. In uncertified regions it applies the worst-case residual inside the confidence bound, so a policy trained there is pessimistic exactly where the evidence is thin.

Safe real-world probing. Each agent follows a mixture of an exploration policy and a known safe fallback controller, in the spirit of the mixture in Provably Safe Sim-to-Real Transfer and the prior-switching of Safe Exploration via Policy Priors. An agent may enter an uncertified local or pairwise region only if the fallback can recover under the worst-case residual. Collisions between agents are handled by decentralized shields of the kind built in Shield Decentralization for Safe Multi-Agent Reinforcement Learning, which need no runtime communication.

Because $N$ agents probing at once multiplies risk, a team-level scheduler hands out a per-step probing budget. Agents whose next state would fall in the most uncertain cells of the shared correction models are favored, so two robots do not spend risk testing the same uncertainty.

Policy learning. The shared policy is trained in the hybrid simulator with a constrained objective: maximize task return subject to a separate cost budget. This follows the reward/cost separation that Quanyi Li et al. argue avoids the "safe but useless" policies produced by hand-tuned penalties. We use a Lagrangian PPO with PID-controlled multipliers as the default learner, since Safety Gymnasium reports that plain Lagrangian updates oscillate. The policy-learning component is deliberately conventional; the contribution is in grounding and probing.

Evaluation settings. None of the files' benchmarks is a multi-agent sim-to-real benchmark, so we build three settings on top of existing ones.

The first is a multi-agent version of the windy 5x5 gridworld from Provably Safe Sim-to-Real Transfer, with 2 to 8 agents. The "real" version adds wind cells and a pairwise push between adjacent agents. Because it is tabular, the factored guarantee can be checked exactly.

The second is the Relative Particle plus Momentum domain from Shield Decentralization for Safe Multi-Agent Reinforcement Learning, scaled to 4 to 32 agents. Its "real" counterpart changes drag and actuator lag and adds a distance-dependent repulsion between agents, similar to rotor downwash. This mirrors the sim-to-sim protocol RGAT used on MuJoCo.

The third is a three-body variant of the particle domain, whose mismatch is deliberately not pairwise, to stress the factored assumption.

If resources allow, a small fleet of identical wheeled robots would serve as a hardware check. The simulation results stand on their own.

Baselines

The naive reference point is a shared policy trained with Lagrangian PPO in the uncorrected simulator and deployed directly, with the decentralized collision shield but no correction of the simulator. It tells us how much the reality gap alone costs. The related sobering number is that Shield Decentralization for Safe Multi-Agent Reinforcement Learning reports unshielded agents averaging over 115 violations per 100 episodes on the momentum task. We expect shielded but uncorrected agents to violate safety mainly through dynamics the shield's model gets wrong.

The strongest existing method is the safe hybrid-model algorithm of Provably Safe Sim-to-Real Transfer. Its reported result is 0% unsafe iterations and 77,958 real samples, against 296,910 for safe learning without a simulator, on the single-agent 5x5 gridworld. No multi-agent result for it is reported in the file. We run it in two ways. The first runs it on the joint state of the team, which is faithful to the method but expected to blow up. The second runs it per agent with independent mismatch tables, which is the obvious decentralized use. Both get the same safe fallback controller, the same real-sample budget, and the same hyperparameter search budget as our method. Beating it means matching its zero-violation record while using clearly fewer total real samples to reach the same certified coverage and final return, with the gap widening as $N$ grows.

A third baseline is unconstrained pooled grounding: RGAT-style learning of an action correction from real multi-agent rollouts collected with no fallback controller, as in Reinforced Grounded Action Transformation. It shows how fast grounding can go when safety is ignored. Our method should approach its sample efficiency with no violations.

A fourth baseline is broad domain randomization over drag, lag, and repulsion strength, combined with the collision shield, trained to the same compute budget.

Three ablations isolate the key ingredients. The first removes the pairwise term $h_\phi$, so only per-agent errors are corrected. The second gives each agent its own unshared correction, which removes pooling. The third replaces the uncertainty-driven probing scheduler with uniform random probing at the same team risk budget.

Why not just…?

Why not just ground the simulator on one robot and copy the correction to all of them? Since the robots are identical, this is the most serious alternative, and for purely per-agent errors it largely does work. Pooling then mainly buys wall-clock speed. It cannot, however, observe pairwise effects, which only appear when robots are close together, and it captures one unit's quirks as if they were universal. The no-pairwise-term ablation combined with single-robot grounding is exactly this baseline. If it matches our method on the domains with interaction-dependent errors, the pairwise contribution is not real and we would narrow the claim to parallel safe grounding.

Why not just use wide domain randomization and a shield? Randomization needs no real data at all. However, Sim-to-Lab-to-Real notes that it gives no guarantee when the real parameters fall outside the randomized range, and the file reports that broad randomization tends to produce overly cautious policies. Our experiments measure both effects. We report the task-return gap at equal violation counts, and we set the "real" repulsion strength just outside the randomized range to test for the silent-failure case.

Why not just add a collision penalty to the reward and fine-tune on hardware? Penalty shaping does not protect the hardware during learning. That is shown by the more than 115 unshielded violations per 100 episodes reported by Shield Decentralization for Safe Multi-Agent Reinforcement Learning, and by the 68.5% unsafe iterations of the unconstrained baseline in Provably Safe Sim-to-Real Transfer. We include penalty-based fine-tuning only as a reference curve in the violation plots.

Why not just use a constrained multi-agent learner like DeCOM in the uncorrected simulator? DeCOM splits each agent's policy into a reward-seeking base and a constraint-fixing correction, and it scales to 500 agents. Its constraints, though, are satisfied with respect to the simulator, not reality. It is complementary: it could replace our Lagrangian learner in the hybrid simulator. We do not treat it as a competitor to the grounding step.

Decisive Experiments

The central experiment plots the real samples needed to reach a fixed certified coverage against team size $N$ on the multi-agent gridworld and the particle domain. Joint-space hybrid grounding should grow steeply with $N$. Per-agent independent grounding should stay roughly flat per agent, so its total grows with $N$. Our pooled symmetric model should keep the total roughly constant, and the time to coverage should fall about as $1/N$. Throughout, we report cumulative real violations, which should be zero for our method, the strongest baseline, and the shielded variants. We also report final task return and cost rate in the corrected real environment.

The second check targets the assumptions. On the three-body domain, the pairwise model should show confidence-bound coverage failures that we can detect. We will report whether the probing rule's safety degrades gracefully, with the fallback controller triggering more often, or whether violations appear. We also introduce variation between robots by perturbing each agent's "real" lag by up to 20%, and check whether the shrinkage-regularized per-agent offset recovers safety and return compared with a purely shared $g_\theta$.

The third check is the single-robot-copy comparison described above. It is the experiment most likely to shrink the claim, and we will report it prominently whichever way it comes out.