When a team of robots is trained in simulation and then deployed, the team rarely gets back the robots it was trained with. One quadruped has a weakened knee motor, another has a slipping wheel, a third has a miscalibrated joint. Multi-robot learning handles this badly. Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning found that shared-policy learning shrugs off sensor errors but collapses under variable actuator errors. Its mean score fell from about −2 to the untrained level of about −6 once 15 or more of 30 agents were affected. Meanwhile, work on adaptive role switching (robots trading jobs such as "explore" and "cover" during a mission) almost always assumes that every robot carries out its assigned job equally well.
We propose to drop that assumption in the same way the imitation-learning literature dropped the assumption that every demonstrator is an expert. Each robot keeps a running, state-dependent estimate of how competently each teammate actually executes each role. It computes this by comparing what a teammate was asked to do with what it achieved. The high-level role selector then gives demanding roles to competent robots and moves degraded ones into roles they can still do. The estimator and the role selector are trained entirely in simulation, under domain randomization (training across many randomly perturbed versions of the simulator). Here the randomization is applied separately to each robot and drifts over time, rather than being shared across the fleet. The target is zero-shot transfer to teams with real, uneven wear.
Core Claim
A role-switching policy that sees explicit, outcome-based competence estimates for its teammates will keep team performance close to the healthy-team level when a minority or even a majority of robots degrade. The same holds when the degradation is of a type never seen in training. Symmetric randomization cannot do this. Heterogeneous randomization with only implicit adaptation, such as a recurrent policy, should also do worse.
The central modeling choice follows Imitation Learning by Estimating Expertise of Demonstrators (ILEED). ILEED infers demonstrator skill offline by treating each observed action as a mixture of the optimal policy and noise, weighted by a state-dependent expertise score. We move this model from offline human demonstrators to online robot teammates. For robot $i$ executing role $k$ in state $s$, let $o$ be the realized outcome of a commanded behavior primitive (for example, displacement achieved or payload height held). We model its probability as
$$p(o \mid s, k, i) = \rho_k(s, \omega_i)\, p_{\text{nom}}(o \mid s, k) + \big(1 - \rho_k(s, \omega_i)\big)\, \eta(o \mid s, k), \qquad \rho_k(s,\omega_i) = \sigma(\langle f_k(s), \omega_i \rangle).$$
In this model, $p_{\text{nom}}$ is the outcome distribution of a healthy robot and $\eta$ is a broad failure distribution. The vector $\omega_i$ is a latent competence embedding for robot $i$, updated online from its recent history. The feature map $f_k$ lets competence depend on both state and role. A robot with torque fade may still be a good explorer on flat ground while being a poor lifter. Because $\rho$ is defined in terms of task outcomes rather than physical parameters, it can register kinds of wear that the randomization never listed. That property is the one we will test hardest.
How This Relates to Prior Work
Two papers motivate the gap. Autonomous and Adaptive Role Selection for Multi-Robot Collaborative Area Search Based on Deep Reinforcement Learning (Zhu et al.) uses a two-level MAPPO hierarchy, in which a centrally trained critic guides robots acting on local observations. A high-level module picks explore or cover and a low-level policy executes the chosen role. It reports 100% exploration and 91.5% coverage with 15 robots in its hardest scene. However, its robots are identical, its tests stay in a 2D grid, and its role reward weights are fixed and hand-tuned. The Zhao et al. study supplies the negative result that frames our problem: symmetric treatment of agents fails under actuator noise, and a "majority rule" determines whether the team recovers. It proposes no mechanism for recovery.
Three lines of work supply the ingredients we reuse. Capability-based partitioning in Approaches to Multi-Robot Exploration and Localization and A Collaborative Aerial-Ground Robotic System for Fast Exploration routes robots to tasks based on known, designed-in differences between platforms. We learn those differences online instead of fixing them at design time. ILEED supplies the expertise-as-mixture model. We apply it online and per teammate, and we avoid its known weakness of needing overlapping state coverage by pooling estimates across roles through the shared feature map $f_k$. Multi-Quadruped Cooperative Object Transport: Learning Decentralized Pinch-Lift-Move (decPLM) supplies the physical testbed and a reward that is not hand-tuned per term: a "constellation" reward based on matching keypoints to anchor sites. Its roles are implicit and staged, it depends on an external planner for initial contacts, and it cannot reassign a robot that slips. We add the explicit reassignment layer on top.
Two further papers constrain the design. Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions infers a partner's role from early interaction with a recurrent predictor. It is the closest analogue to our estimator, but it infers what a partner intends, not what the partner is physically able to do, and it is tested only in grid games. The VMAS paper, VMAS: A Vectorized Multi-Agent Simulator for Collective Robot Learning, shows that shared-parameter MAPPO fails to break symmetry in its "Give Way" scenario, while HetIPPO (a variant with a separate policy per agent) succeeds. This warns that heterogeneity has to enter the policy's inputs explicitly.
Manual reward engineering enters as a constraint rather than as the core mechanism. We will not hand-tune role weights. The role selector is trained only on the team task reward: coverage for search and constellation-plus-delivery for transport. We will test whether competence conditioning removes the weight sensitivity that Zhu et al. report.
Research Program
The first component is a heterogeneous, time-varying randomization scheme. Each robot independently draws a degradation trajectory: actuator gain fade, output noise of the OV type from the Zhao et al. study, fixed calibration offsets, and speed caps. Onset times are random, and the fraction of degraded robots ranges from zero to a majority. Some degradation families are held out entirely for testing.
The second component is the competence estimator. It takes a window of (commanded primitive, realized outcome) pairs for a teammate and fits the mixture likelihood above. It is trained without labels on the degradation parameters, so it cannot simply read off the randomization settings.
The third component is a role selector trained with MAPPO whose inputs include teammates' $\rho_k$ values. It sits above fixed behavior primitives, in the spirit of Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks. MAPLE's primitive layer reached 90% average success in simulation, against 19% for flat control, and transferred zero-shot to a real arm. For transport, the roles are load-bearing pinch, stabilizing contact, and scout/guide.
We will run this in two domains. The first is area search in VMAS, reproducing Zhu et al.'s explore/cover hierarchy with degraded agents. The second is decPLM-style transport in IsaacLab with two to six Unitree Go2 robots, followed by physical deployment.
Baselines
The naive reference point is Zhu et al.'s hierarchy with homogeneous training and no degradation modeling. It establishes how much degradation hurts. On healthy teams it should match the published 100% exploration and 91.5% coverage (15 robots, Super hard). As far as the file shows, no degraded-team result exists for it; Lacuna returned no reported result for the degraded setting.
The second baseline is standard symmetric domain randomization, in which every robot receives the same perturbation distribution. It represents the default practice from Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. The Zhao et al. result, where the score drops from about −2 to about −6 under OV15/OV25 on NoisyKukaReacher, predicts it will fail once a majority of robots degrade. We will reproduce that setting as a sanity check.
The strongest competitor is heterogeneous per-robot randomization with a recurrent role selector and separate per-agent policies (HetIPPO style). This policy sees the same training distribution as ours and can adapt implicitly from its history, with no explicit competence estimate. All three learned systems will share simulator steps, network size, primitive library, and a matched hyperparameter search budget.
Two ablations isolate the key ingredient. The first swaps the unsupervised outcome-mixture estimator for supervised regression of the true degradation parameters, which uses privileged labels in simulation. The second feeds the true $\rho$ as an oracle, giving an upper bound. We count our method as beating the strongest baseline if it recovers significantly more of the healthy-team gap on held-out degradation types across seeds. Matching it only on seen types does not count.
Why not just…?
Why not just use heterogeneous randomization with a recurrent policy and let adaptation stay implicit? This is the real threat. On degradation types seen in training, it may match us. Our bet is that an explicit, outcome-defined $\rho$ generalizes better to wear types that were never simulated and to team sizes never seen in training, because the role selector reads a small, fixed-meaning signal rather than raw histories. The held-out-degradation experiment settles this. If the recurrent baseline matches us there too, the explicit estimator is unnecessary and we will report that.
Why not just threshold each robot's tracking error and demote it by rule? A hand-set rule reintroduces the manual tuning we are trying to remove. It also cannot tell a hard state from a bad robot, which is the confusion ILEED's state-dependent formulation exists to resolve: slow progress on rough terrain is not a fault. We will run a tuned threshold rule as an additional baseline. The state-dependence claim is tested by degradations that matter in only some states, such as torque fade that matters only under load.
Why not just regress the physical degradation parameters directly? This is ablation one. Supervised regression is simpler and may be more accurate on known degradation families. It cannot, however, represent a family it has no output for. We expect it to win on seen types and lose on held-out ones.
Proposed Experiments
In VMAS search, we sweep the fraction of degraded robots (0, 1/3, 1/2, 2/3) with 4, 8, and 15 robots. We report exploration percentage, coverage percentage, and time to 90% exploration, following Zhu et al. We also report how often degraded robots end up in low-demand roles, which checks that reassignment is actually happening. On NoisyKukaReacher, we add a two-role variant (reacher versus spotter) to test whether role reassignment breaks Zhao et al.'s majority rule.
In IsaacLab transport, we report drop rate and success rate. decPLM reports a drop rate falling from 5% to under 1% as healthy teams grow toward 10 robots; that is our healthy-team reference. The central test degrades one or two robots in the middle of a transport.
The decisive checks come in three parts. The first is the held-out-degradation comparison against the recurrent heterogeneous baseline. The second compares the correlation between estimated $\rho$ and true degradation when the estimator is trained with and without labels. The third is physical deployment on Go2 robots, where one or two units receive a software-imposed torque cap and injected joint offsets, a controllable and honest stand-in for wear. There we measure zero-shot transport success against the symmetric-randomization policy over repeated trials. A final check retrains with deliberately perturbed role reward weights; flat performance would support the claim that competence conditioning, not weight tuning, is doing the work.
References
- Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning
- Imitation Learning by Estimating Expertise of Demonstrators
- Autonomous and Adaptive Role Selection for Multi-robot Collaborative Area Search Based on Deep Reinforcement Learning
- Approaches to Multi-Robot Exploration and Localization
- A Collaborative Aerial-Ground Robotic System for Fast Exploration
- Multi-Quadruped Cooperative Object Transport: Learning Decentralized Pinch-Lift-Move
- Role play: Learning adaptive role-specific strategies in multi-agent interactions
- VMAS: A Vectorized Multi-agent Simulator for Collective Robot Learning
- Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots