Joint Worst-Case Physics for Heterogeneous Robot Teams

When a team of different robots, such as a heavy wheeled base and a nimble manipulator, is trained in simulation and then deployed, the failures that matter most are often not one robot's simulator being slightly wrong. They are mismatches that interact. A few milliseconds of extra motor delay on the slow robot, combined with lower wheel friction on the fast one, can break a coordination pattern that survives either error alone. Today's standard defenses randomize each physical parameter independently, or make each agent individually cautious. Both approaches can miss these correlated combinations, which may sit in small corners of a large continuous parameter space.

We propose to treat the reality gap for a heterogeneous team as a game between the team and "nature." Nature picks a joint vector of physical parameters covering every robot's friction, mass, latency and actuator gains. The team picks continuous control policies. This game has a continuous strategy space for nature, but results on continuous games say that its worst case can be captured by a mixture over a few specific parameter settings. We will find those settings with a double-oracle loop, a procedure that alternates between training the team against the current finite set of hard physics settings and searching the continuous space for a new setting that hurts the team most. We add a feasibility test so nature cannot win by proposing physics under which no team could succeed. The deliverable is a trained team and a compact stress-test set of a few physics settings that explains where the sim-to-real risk concentrates.

Core Claim

The claim is empirical and falsifiable. For heterogeneous teams, the worst-case reality gap lives in correlated, cross-agent parameter combinations. A shared joint adversary over the continuous physics space, with finite support and a feasibility constraint, finds those combinations and trains against them. Two families of methods cannot do this: uniform domain randomization over the same ranges, and per-agent pessimism. If the claim is right, the joint adversary should beat both on held-out worst-case return and on transfer to unmodeled effects, while giving up little nominal performance. If uniform randomization matches it, the claim is wrong and the finite-support set is only a diagnostic.

Written out, the team policy profile $\pi = (\pi_1,\dots,\pi_N)$ and nature's distribution $q$ over the joint parameter box $\Theta \subset \mathbb{R}^d$ play

$$\max_{\pi} \; \min_{q \in \Delta(\Theta_\lambda)} \; \mathbb{E}_{\theta \sim q}\big[J(\pi,\theta)\big], \qquad \Theta_\lambda = \{\theta \in \Theta : J^*(\theta) \ge \lambda\},$$

where $J(\pi,\theta)$ is team return under physics $\theta$, $J^*(\theta)$ is the best return any team can reach under $\theta$, and $\lambda$ is a performance threshold. The set $\Theta_\lambda$ keeps only parameter settings where success is possible.

Where This Sits

The proposal takes one method from each of three lines of work and applies it to a setting none of them address.

The motivation comes from Distributionally Robust Cooperative MARL with Value Factorization. That paper proves that giving each agent its own worst-case model breaks the rule that individually greedy actions should compose into the team-optimal action. Its fix is to tie every agent to one shared global adversary. It works only with discrete-action value factorization (VDN, QMIX, QTRAN), and its uncertainty is an abstract contamination of transition probabilities rather than physical parameters. We carry the "shared adversary" insight to continuous-action heterogeneous teams, where the adversary is a physical parameter vector. The same paper also reports that robustness improved nominal performance in cooperative settings: robust QTRAN gained 10–40% average reward on SustainGym BuildingEnv. Whether that holds under contact-rich continuous physics is one of our questions.

The game-solving machinery comes from Double Oracle Algorithm for Computing Equilibria in Continuous Games. That paper grows a finite restricted game with continuous best responses and proves that finite-support approximate equilibria exist. Its main weakness is that local best-response search can stop too early. We inherit this weakness and address it with restarts, and with a separate evaluation adversary so that false convergence is caught rather than rewarded.

The guard against over-pessimism comes from Feasible Adversarial Robust RL. That work uses population-based game solving (PSRO) against an environment adversary and penalizes the adversary for proposing unsolvable tasks. It is the closest existing work, but it controls a single agent. It therefore cannot express the cross-agent correlations that are our subject, and it never asks whether a joint adversary differs from a factorized one.

Two single-agent sim-to-real methods set the bar for safety and pessimism. SPiDR penalizes disagreement across an ensemble of randomized dynamics models, which gives zero-shot safety on a Go1 quadruped and a RaceCar. SR² resets an imperfect simulator to real anchor states. Neither handles teams.

Two multi-robot papers show that team sim-to-real is feasible but rely on heuristics or low-level control. SeqWM transfers two identical Go2-W robots using randomized communication dropout. Decentralized aerial manipulation transfers identical quadrotors by delegating robustness to a fast analytic inner loop. Our focus on heterogeneous platforms is the regime both leave open.

The team learner is HASAC, a maximum-entropy method that updates heterogeneous agents one at a time. We use it unchanged as the team's best-response oracle, so any gain is attributable to what the team trains against, not to a better learner.

Research Program

The loop maintains a finite set of physics settings $\{\theta_1,\dots,\theta_k\}$ with mixture weights, and a population of team policies. Each iteration has four steps.

First, a meta-game of estimated returns between team policies and physics settings is solved by a small linear program. This gives the current worst-case mixture over physics and the team's current mixture over its policies.

Second, HASAC trains a new team policy against the current physics mixture, with a fixed step budget.

Third, nature's best response is found by global search over the continuous joint box $\Theta$. We use CMA-ES with multiple restarts, seeded partly at perturbations of the existing support points.

Fourth, a candidate $\theta$ is admitted only if it passes a feasibility check. The current team is fine-tuned at that $\theta$ for a small fixed budget, and the check asks whether that fine-tuned specialist reaches the threshold $\lambda$. This is a cheaper proxy for $J^*(\theta)$ than the full specialist training in FARR. The proxy is conservative in one direction: it may wrongly reject hard-but-solvable physics. We will measure how often this happens by occasionally training full specialists.

The loop stops when the estimated gap between the restricted-game value and nature's best response falls below a tolerance, or when the budget runs out.

Two analyses accompany the method. The first is a support audit. For each task we record how many physics settings the final mixture needs, and whether those settings are cross-agent correlated or axis-aligned. Axis-aligned means only one agent's parameters are pushed to an extreme. This directly tests whether correlated failures are real. The second audit re-runs the loop with a factorized adversary, which must choose each agent's parameters independently. The gap between joint and factorized adversaries measures how much the heterogeneous coupling itself contributes.

Baselines

The naive reference is HASAC trained at nominal physics with no randomization. It shows the size of the gap we are trying to close.

The strongest practical competitor is HASAC with uniform domain randomization over exactly the same joint box $\Theta$. Uniform randomization is the standard sim-to-real recipe (OpenAI et al.) and it already covers every correlated combination in principle. It gets the same total environment steps as our full loop, including the steps spent on nature search and feasibility fine-tuning. It also gets the same hyperparameter-tuning budget, tuned on the same validation perturbations.

The second strong competitor is HASAC with a SPiDR-style ensemble-disagreement penalty applied per agent. This represents the "make each agent pessimistic" route. The relevant published numbers are single-agent: SPiDR reports 100% real-world constraint satisfaction, and standard randomization showed 26.1x higher safety violations on Go1 and 6.2x on the RaceCar. No reported result was found on Lacuna for any multi-agent or heterogeneous version. Its penalty weight gets the same tuning budget as our $\lambda$.

We also run an adversarial baseline that trains the team against a single, non-mixed nature best response each round, in the style of robust adversarial RL. This isolates the value of maintaining a finite mixture. For HASAC itself, the file reports best results in 31 of 35 tasks and useful Bi-DexHands policies in 10M steps where MAPPO learned nothing. No reported per-task MAMuJoCo return or robustness number was found on Lacuna for HASAC.

Two ablations isolate the key ingredients: the factorized adversary, which removes cross-agent correlation, and the loop with feasibility turned off, which removes the unsolvable-physics guard.

Beating the baselines means two things. Our method must have higher worst-case return than both uniform randomization and per-agent pessimism under the held-out evaluation adversary, across at least five seeds, with non-overlapping confidence intervals. It must also keep nominal return within a pre-registered margin of uniform randomization. On hardware, it must have fewer failed or colliding trials than uniform randomization on the same episodes.

Why not just…?

Why not just use uniform domain randomization over the same ranges? This is the real threat. Our case against it is only a hypothesis: correlated worst cases occupy small volume, so uniform sampling rarely visits them, and averaging over the whole box spends capacity on easy settings. If the support audit finds that the worst cases are axis-aligned, or if uniform randomization matches our worst-case return at equal compute, the method adds nothing beyond a diagnostic. We will report that outcome as the answer.

Why not just make each agent individually pessimistic, SPiDR-style? The DrIGM paper proves that independent per-agent robustification misaligns decentralized actions from the robust team optimum, at least for discrete factorized values. Per-agent ensemble variance also cannot distinguish "my own dynamics are uncertain" from "my partner's dynamics make our joint action risky." The per-agent-pessimism baseline and the factorized-adversary ablation test this directly in the continuous setting. A small gap would mean the theory's concern does not bite in practice.

Why not just discretize the parameter space and enumerate it? With several parameters per robot across heterogeneous agents, a grid grows exponentially in the number of parameters, and extremes are resolved only at grid resolution. Both are standard objections to discretizing continuous games (Tammelin et al.). In the smallest two-agent task, where a coarse grid is affordable, we will run an exhaustive grid evaluation as a check that our found support points are real worst cases rather than artifacts of the search.

Why not just push robustness into a strong low-level controller? The aerial manipulation work shows an analytic inner loop can absorb aerodynamic mismatch for identical quadrotors. That inner loop corrects each robot's own tracking error. It does not change the high-level coordination that fails when partners respond at different speeds, so we include a heterogeneous task where the failure comes from timing between robots.

Proposed Experiments

The simulation study uses Multi-Agent MuJoCo, where joints of one body are split across agents with different actuator groups. We perturb per-agent-group actuator gain, joint damping, friction and action latency. We also use Bi-DexHands, perturbing per-hand latency, finger friction and object mass. Last, we build a heterogeneous version of SeqWM's PushBox task: one heavy slow robot paired with one light agile robot, in the spirit of the Khepera and Magellan setting of Safety Barrier Certificates.

The primary metric is worst-case return under a separate evaluation adversary. This adversary is a fresh CMA-ES run with ten times the search budget and different seeds, never used in training, which avoids scoring our method with its own oracle. Secondary metrics are mean return over held-out uniform perturbations, nominal return, and total environment steps.

To test misspecification, every method is also evaluated on effects deliberately left outside $\Theta$, namely gear backlash and sensor bias. This checks that the method does not only shine inside its own model, echoing the realizability caveat in Provable Sim-to-real Transfer.

The decisive checks come in three parts. First, whether the joint adversary beats the factorized one and uniform randomization in worst-case return. Second, whether the final support sets are dominated by cross-agent correlated points. Third, whether nominal return holds up, which tests whether the no-robustness-penalty result from DrIGM survives continuous contact physics.

If simulation results are positive, a hardware phase deploys the heterogeneous PushBox team, a Go2-W paired with a smaller wheeled robot, under motion capture. It compares our policy with uniform randomization and per-agent pessimism over matched trials, counting task success and collisions. The physics settings in the support set are used to choose the physical conditions tested, such as added payload on the heavy robot and a low-friction floor for the light one.