Transformer policies for multi-robot teams can be trained with four robots and deployed with sixteen. They do this by reading every robot and obstacle as one token in a sequence, so the same weights handle any number of tokens. Safety does not carry over so easily. The standard way to make a team safe is a constrained objective with one team-level cost budget and one penalty multiplier, and both are tied to the team size used in training. With four times as many robots there are many more close encounters per episode. The budget tuned at four robots no longer means anything, and the multiplier is wrong for every robot at once. Engineers end up re-tuning collision penalties for each team size, which is the manual reward engineering this line of work was supposed to remove. Separately, the safety estimates learned in simulation drift on hardware, often because of interactions between pairs of robots that the simulator models poorly.
This note proposes making the safety side of an attention policy as size-independent as the policy itself. Each robot gets its own cost budget, chosen once in physical units (for example, near-miss exposure per robot per episode). Each robot's expected future cost is predicted as a sum of pairwise contributions from its neighbours, not as a softmax-weighted average. A safety penalty weight, computed from the robot's local situation, replaces the single global multiplier. The bet is that one budget and one set of hyperparameters, chosen at the training team size, then hold at four times that size without re-tuning. A second bet is that a sum of pairwise terms can be re-calibrated on a small amount of real data, because every real episode contains many pairwise interactions.
Core Claim
Safety for one robot is local: it depends on who is near it, not on how many robots exist. A constrained multi-agent objective should therefore be written per robot. Let $c_{ij}(s,a)$ be the cost from the interaction between robot $i$ and entity $j$, such as a graded near-miss penalty that rises as their separation shrinks. Let $\gamma$ be the discount and $\epsilon$ a single per-robot budget that does not depend on team size $N$. The problem is
$$\max_\pi \; \mathbb{E}\Big[\sum_t \gamma^t r(s_t,a_t)\Big] \quad \text{s.t.} \quad \mathbb{E}\Big[\sum_t \gamma^t \sum_{j \neq i} c_{ij}(s_t,a_t)\Big] \le \epsilon \;\; \text{for every robot } i.$$
The architectural claim is about how the cost critic (the network that predicts robot $i$'s future safety cost) pools over neighbours. Standard attention takes a softmax-weighted average of per-neighbour features. Averages dilute: one dangerous neighbour among many harmless ones contributes roughly $1/N$ of its risk, unless the learned attention scores separate sharply. Scores calibrated at the training size need not do that at larger sizes. We instead compute
$$Q_c^i(s,a) = \sum_{j \in \mathcal{N}(i)} \phi_\theta(h_i, h_j, a_i),$$
where $h_i, h_j$ are the transformer's contextual embeddings of robot $i$ and entity $j$, $\mathcal{N}(i)$ is the set of entities in robot $i$'s view, and $\phi_\theta$ is a small network that outputs a nonnegative pairwise cost. Context still flows through attention inside the encoder, so $h_i$ and $h_j$ know about the crowd. The final pooling is additive, so adding a harmless neighbour adds near zero and adding a dangerous one adds its full risk.
The penalty weight $\lambda(h_i)$ comes from the same encoder. It is trained by dual ascent on each robot's constraint violation, so robots in dense clusters are pushed harder toward safety than robots in open space.
Two further claims follow from this structure. The pairwise term $\phi_\theta(h_i,h_j,a_i)$ is an explicit, inspectable "how much does $j$ endanger $i$" quantity. That gives an operational answer to the open question in Provably Safe Online Multi-Agent Navigation in Unknown Environments, whose graph-attention network selects relevant neighbours but whose authors note that no formal link exists between attention weight and constraint importance. Second, sim-to-real error in safety is often itself pairwise, such as unmodeled aerodynamic interference between nearby drones. Fine-tuning only $\phi_\theta$ on real logs therefore uses every pair in every episode as a training sample.
Relation to Prior Work
Three lines of work motivate the gap.
Attention-based scaling shows that entity-as-token policies transfer across team sizes. UPDeT matches action groups to observed entities and reports transfer "10 times faster" from 3 to 5 agents. WW2C trains a transformer communication policy on 12 quadrotors, runs it zero-shot on 24, and flies it on Parrot Bebop 2 hardware. Neither optimises a constrained objective: WW2C delegates safety to an external model-predictive planner, and UPDeT has no safety term at all.
Safe multi-agent RL supplies constraints but not scaling. Multi-Agent Constrained Policy Optimisation gives MACPO and MAPPO-Lagrangian with near-zero costs on Safe Multi-Agent MuJoCo. It uses fixed-size networks, a team-level constraint, and sequential per-agent updates. DeCOM splits each policy into a reward-seeking base and a constraint-reducing perturbation and scales to 500 agents, but over fixed communication graphs. MOSDT is the closest existing work. It is a parameter-shared decision transformer that replaces scalar cost targets with binary safe/unsafe tokens. It is purely offline and uses fixed agent slots per task. We keep its lesson that finely tuned scalar budgets are fragile, and our physical-unit per-robot budget is an online analogue of it. We add what it lacks: online exploration and variable team size.
Safety filters constrain the design. Probabilistic Safety Barrier Certificates and Safe MARL via Shielding give hard guarantees under a model or a discrete abstraction. The first paper documents deadlock in dense scenes when worst-case bounding volumes are used. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that safer policies collect fewer violations, which starves cost estimation. That negative result is why we use graded near-miss costs rather than binary collisions.
On sim-to-real, Provably Safe Sim-to-Real Transfer localises where the simulator is wrong and explores there under a safe mixture policy, reporting "0% unsafe iterations" against "68.5%" for unconstrained transfer. It does so only for tabular, single-agent problems. Sim-to-Lab-to-Real uses a reachability-based backup policy and PAC-Bayes certificates on one quadruped, with a hand-set intervention threshold. Our recalibration step borrows the idea of correcting the model only where it is wrong. The correction is applied to a pairwise cost head in continuous space, and we make no claim of a formal certificate.
Research Program
The work has three stages.
The first stage is a diagnostic study that tests the dilution hypothesis. We train a standard softmax-pooled attention cost critic at $N=4$ on multi-agent navigation. We then measure its calibration (predicted versus realised discounted cost) as $N$ grows to 8, 16 and 32. We also measure it at fixed $N$ as we add many distant, harmless entities. If the critic's underestimation grows with crowd size, the mechanism is real. If it does not, the core claim weakens, and the remaining stages reduce to an engineering comparison.
The second stage builds the method. A shared transformer encoder over agent and obstacle tokens feeds three heads: the actor, a reward critic, and the sum-pooled pairwise cost critic. A state-dependent multiplier head is trained by projected dual ascent on each robot's own constraint. To damp the oscillations that Safety Gymnasium documents for plain Lagrangian updates, we add the PID damping used in its CPPO-PID variant. The task reward is deliberately minimal: progress toward one's own goal plus a goal-reached bonus. All collision handling lives in the constraint. One budget $\epsilon$ is set in physical units and kept fixed across every team size and every experiment.
The third stage tests transfer to a mismatched "real" world. The primary setting is sim-to-sim. The training simulator omits a pairwise effect, and the deployment simulator adds one: separation-dependent downwash and actuation latency, a continuous analogue of the windy cells in the gridworld of Provably Safe Sim-to-Real Transfer. We collect a fixed small budget of deployment episodes under a conservative behaviour policy and fine-tune only $\phi_\theta$ and the multiplier head. We then measure violations and task success.
If hardware is available, the same protocol runs on a small quadrotor fleet in the style of WW2C. This is a stretch goal and the proposal's conclusions do not depend on it.
Baselines
Naive penalty. The do-nothing reference is the same transformer policy trained with unconstrained PPO plus a fixed collision penalty. The penalty weight is tuned at $N=4$ and frozen. This is the manual-engineering status quo. Quanyi Li et al. document its two failure modes: penalties that are too weak produce reckless policies, and penalties that are too strong produce robots that stand still. Beating it means fewer violations at every $N$ without lower success at the training size.
Strongest existing method. The strongest competitor is MAPPO-Lagrangian from Multi-Agent Constrained Policy Optimisation, run with the official codebase (MACPO code). Its fixed-size network is replaced by our transformer encoder so that it can run at all $N$; this isolates the objective and pooling from the architecture. It is the strongest competitor because it is the established first-order constrained multi-agent method and is reported to achieve "near-zero costs" on Safe Multi-Agent MuJoCo. The file gives no per-task numbers for that benchmark and no reported result on Lacuna for MAPPO-Lagrangian at varying team size. For single-agent context, Safety Gymnasium reports PPO-Lag with a "98% reduction in safety violations" and "only a 45% drop in total reward" relative to PPO on velocity tasks. Fairness means the same encoder size, environment steps, seeds (at least five), and the same hyperparameter search budget, spent only at $N=4$. We run it in three variants that give its team-level budget every chance to scale: budget fixed, budget scaled with $N$, and budget scaled with expected pair count.
Safety-filter baseline. An attention policy trained without a constraint is wrapped at execution by a probabilistic barrier-certificate quadratic program, following Probabilistic Safety Barrier Certificates. That paper reports it "avoided all crashes" under bounded noise at under 2 ms per robot for 11 quadrotors. It is the strongest competitor on raw violations in the matched-model setting.
Ablations. Two ablations isolate the key ingredients. The first keeps everything but uses softmax pooling in the cost critic. The second keeps sum pooling but uses one global multiplier.
What beating the baselines means. At $4\times$ the training size, without re-tuning, the method keeps each robot's realised cost within $\epsilon$ in the large majority of seeds and episodes. It does so while matching or exceeding the goal-reaching rate of every baseline that also satisfies the budget.
Why not just…?
Why not just scale the team-level budget with $N$ in MAPPO-Lagrangian? This is the most serious threat, and it is why the main comparison includes two rescaled variants. Rescaling fixes the average violation level. It cannot fix the distribution: a single global multiplier pushes equally on a robot in a dense cluster and a robot alone in open space. The prediction is that rescaled MAPPO-Lagrangian satisfies the team budget on average while robots in clusters exceed their share and isolated robots are over-conservative. We test this in scenes with deliberately mixed density, measuring per-robot rather than team-average violations. If rescaled MAPPO-Lagrangian matches our per-robot numbers there, the global-versus-local multiplier part of the proposal is unnecessary, and we will report that.
Why not just put a barrier-function safety filter on an unconstrained attention policy? Filters give strong guarantees when the model is right. They have two documented weaknesses: deadlock in dense scenes, noted in the barrier-certificate paper itself, and degradation under unmodeled dynamics. A filter-wrapped policy also never learns to avoid situations that trigger the filter, so its task progress drops as crowds grow. We expect the filter to win on violations in the matched-dynamics setting. The decisive comparisons are deadlock rate at high $N$ and violation rate under the deployment-simulator mismatch, where the filter's model is wrong and only our pairwise head is re-fitted.
Why not just use a distance cutoff and sum hand-designed proximity costs? A cutoff radius is size-invariant by construction and needs no learned critic. WW2C reports that its learned transformer cut communication by "70% to 80%" relative to full broadcasting with "often less than a 0.4% difference" in collision rate. The paper attributes this to distance rules over-reacting to nearby robots moving apart and missing fast distant head-on approaches; the file reports no separate number for the distance heuristic itself. We include a cutoff-plus-analytic-cost critic as an extra ablation. If it matches the learned pairwise critic in head-on and crossing scenarios, the learned $\phi_\theta$ is not earning its place.
Proposed Experiments and Decisive Checks
The main testbed is variable-size multi-agent navigation. We use the Corridor, Formation and Line tasks from Safe Continuous-time MARL via Epigraph Form (no exact numbers reported in the file) and the multi-agent navigation tasks in Safety Gymnasium. All are modified so that team size can be varied: train at 4, test at 4, 8, 16 and 32. A quadrotor position-swap scenario modelled on WW2C's 12-to-24 setting gives a second, denser regime.
Metrics are reported per robot: the fraction of robots exceeding $\epsilon$, cost-critic calibration error, goal-reaching rate, time to goal, and deadlock rate. All are reported as functions of $N$ and of local density.
The first decisive check is the dilution diagnostic from stage one. Softmax-critic calibration error should grow with $N$ and with the number of distractor entities, while the sum-pooled critic's error stays flat. The second is the transfer curve. Per-robot violation rate versus $N$ should stay flat for our method under one frozen budget and rise for the softmax ablation and fixed-budget MAPPO-Lagrangian. The third is the sim-to-sim recalibration curve: deployment-world violations as a function of the number of deployment episodes used to fine-tune $\phi_\theta$. Here we expect a steeper drop than fine-tuning a monolithic cost critic on the same data, because each episode contributes many pairwise samples.
Failure at the first check refutes the mechanism. Failure at the second against rescaled MAPPO-Lagrangian refutes the need for local multipliers. Failure at the third means the pairwise structure does not help close the reality gap, and the proposal shrinks to a scaling result in simulation alone.
References
- Provably Safe Online Multi-Agent Navigation in Unknown Environments
- UPDeT: Universal Multi-agent RL via Policy Decoupling with Transformers
- Learning scalable and efficient communication policies for multi-robot collision avoidance
- Multi-Agent Constrained Policy Optimisation
- DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
- MOSDT: Self-Distillation-Based Decision Transformer for Multi-Agent Offline Safe Reinforcement Learning
- Multi-Robot Collision Avoidance under Uncertainty with Probabilistic Safety Barrier Certificates
- Safe Multi-Agent Reinforcement Learning via Shielding
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Provably Safe Sim-to-Real Transfer
- Sim-to-Lab-to-Real: Safe reinforcement learning with shielding and generalization guarantees
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Safe Continuous-time Multi-Agent Reinforcement Learning via Epigraph Form