Neighborhood-Factored Discrepancy Inference for Multi-Robot Sim-to-Real

When a policy trained in simulation is deployed on real robots, each robot is wrong about the world in its own way. One drone's motors are weaker than modeled, another carries a heavier battery, a third has more actuator lag. In a team these errors multiply. If each of $N$ robots can be in one of $K$ discrepancy "types", the team as a whole can be in $K^N$ configurations. This is the same exponential blow-up that makes the space of joint actions intractable in multi-agent reinforcement learning (RL), now appearing in the space of possible realities rather than the space of choices. Today's standard remedy is domain randomization: train while randomly perturbing simulator parameters so the policy survives many worlds. That remedy asks every agent to hedge against the whole product space at once. We argue this is one plausible cause of the "paranoid", overly slow behavior reported when multi-robot policies are trained with heavy randomization.

We propose that each robot should infer, while it runs, a compact belief about its own discrepancy type and the types of its immediate neighbors in the interaction graph. Neighbors share these beliefs as short messages, and actions are conditioned on them. Nobody reasons about the full joint configuration. The approach borrows the structural move that tames joint action explosion, locality in the interaction graph, and applies it to the uncertainty over physics. It predicts that adaptation cost grows linearly with team size. It also makes a sharp, falsifiable claim: in tightly coupled tasks such as carrying a shared payload, knowing your neighbors' discrepancies matters beyond knowing your own. In loosely coupled tasks it should not.

Core Claim

Multi-robot sim-to-real transfer has two problems that are usually studied separately. The first is the joint uncertainty set over per-robot dynamics, which grows exponentially in $N$. The second is the conservatism that uniform randomization induces. We claim both can be handled by factoring the uncertainty along the same neighborhood structure that makes localized multi-agent learning scale.

Concretely, agent $i$ acts with a policy $\pi_i(a_i \mid h_i, \{b_j\}_{j \in \mathcal{N}_\kappa(i)})$. Here $h_i$ is agent $i$'s own observation-action history. $b_j$ is a belief, a probability vector over $K$ discrepancy types, that agent $j$ estimates for itself and broadcasts. $\mathcal{N}_\kappa(i)$ is the set of agents within $\kappa$ hops of $i$ in the physical coupling graph, including $i$. Per-agent inference and communication then cost $O(K\,|\mathcal{N}_\kappa|)$ rather than $O(K^N)$.

The decisive empirical prediction is a crossover. Self-only inference, where $\kappa=0$ and each robot identifies its own dynamics, should suffice when agents are weakly coupled. Neighbor inference with $\kappa \ge 1$ should win when a teammate's hidden weakness changes what you should do. Full-team inference should add little beyond $\kappa=1$ while costing more data and bandwidth.

Grounding in Prior Work

Four papers motivate the gap.

Transferring Multi-Agent Reinforcement Learning Policies for Autonomous Driving using Sim-to-Real moves a policy trained with MAPPO (multi-agent PPO with a centralized critic) from a digital twin to three physical Duckiebots. It finds a "Goldilocks" effect. Without randomization the policy fails on hardware, heavy randomization makes it drive too slowly to earn reward, and medium randomization works. The randomization levels were chosen by hand, the same across agents, and tested only at $N=3$. We keep its hardware protocol and replace fixed randomization with per-agent, neighbor-shared inference.

Sample-Efficient Distributionally Robust Multi-Agent Reinforcement Learning via Online Interaction formalizes model mismatch in games with worst-case uncertainty sets. It notes that one agent's mismatch can cascade through teammates' adaptation, and it proves sample complexity scaling with the full joint action space. That paper supplies the diagnosis: worst-case reasoning over the joint space is both expensive and conservative. Its tests are 2-agent synthetic games. We do not pursue worst-case guarantees; we pursue inference over which world the team is in.

Bayesian Robust Cooperative Multi-Agent Reinforcement Learning Against Unknown Adversaries (BATPAL) supplies part of the method. It partitions disturbances into $K$ severity brackets, defined by damage to a reference policy. A recurrent belief network then lets defenders infer the bracket online, which preserves nominal performance where minimax training does not. BATPAL models one adversary, and it names the combinatorial explosion of arbitrary perturbed subsets as unscalable. That explosion is exactly what our per-agent, neighborhood-local beliefs are designed to avoid. We also apply the bracketing to physical parameters rather than adversarial policies.

Scalable Reinforcement Learning of Localized Policies for Multi-Agent Networked Systems supplies the structural justification. It shows that distant agents' influence decays exponentially with graph distance, so critics truncated to $\kappa$-hop neighborhoods lose little. That result concerns nominal dynamics. Our hypothesis is that the same decay applies to the influence of distant agents' hidden physical discrepancies, and this is what licenses truncating the belief exchange.

The mean-field methods, Learning Mean-Field Games and Pessimism Meets Invariance, constrain the design. They assume interchangeable agents, and heterogeneous per-robot discrepancy is precisely what breaks that assumption. So we do not summarize teammates by a population average. Tesseract shows that low-rank structure can tame joint action spaces. We use graph locality instead of low rank because physical coupling in robot teams is local and known in advance.

The single-agent recipe of What Matters in Learning A Zero-Shot Sim-to-Real RL Policy for Quadrotor Control? (SimpleFlight) is adopted wholesale. It combines rotation-matrix inputs, a time-aware critic, smoothness penalties, and system identification for measurable parameters with randomization only for sensitive ones. This ensures any gains we report are multi-agent gains, not missing single-agent hygiene.

Research Program

We build on OmniDrones, whose GPU simulator sustains over 310,000 frames per second for 4-agent cooperative tasks. Its tasks already include formation flight and suspended-payload transport under low-level rate or rotor control.

During training, each drone draws its own physical parameters independently: thrust coefficient, motor time constant, mass offset, and a sensor-latency term. Each parameter vector is mapped to one of $K$ types by bracketing its effect on a reference policy's return, following BATPAL's partitioning. A per-agent recurrent encoder reads $h_i$ and outputs $b_i$. It is trained with a supervised loss against the privileged type label, alongside the usual RL loss.

Agents exchange $b_i$ with graph neighbors. The coupling graph is given by tether or proximity: the rope network for payload transport, nearest neighbors for formation flight. The critic sees privileged true parameters but is truncated to $\kappa$-hop neighborhoods, following the localized actor-critic design, so it scales with $N$.

Four analyses accompany the method. First, we measure how the optimal fixed randomization width moves as $N$ grows. We hypothesize it must shrink, because the joint uncertainty set inflates, and this would explain why hand-tuned "medium" randomization does not transfer across team sizes. Second, we measure how much of each agent's value depends on distant agents' types, as an empirical check of the decay assumption. Third, we evaluate robustness to message dropout and delay. Fourth, we test how beliefs behave when reality falls outside all $K$ brackets.

Baselines

The naive reference points are MAPPO trained with no randomization and MAPPO with uniform, identical randomization at the "medium" level. This mirrors the Duckietown study, whose medium-randomization MAPPO earned a real-world reward 1.85 times that of the Gipps rule-based lane-changing model (Transferring Multi-Agent Reinforcement Learning Policies...). That is the one hardware number available for a multi-agent sim-to-real baseline in this file. On our Duckietown replication, beating it means exceeding that baseline's real-world reward under the same Gipps-normalized protocol. The medium width will be re-tuned separately for each team size with the same search budget our method receives, so the baseline is not handicapped by a stale setting.

The strongest competitor is MAPPO with the full SimpleFlight recipe plus per-agent self-inference ($\kappa=0$). In this variant each drone infers its own type from its own history and nothing is shared. It is strongest because it already captures most of what is known to matter for single-robot transfer, and it scales trivially. SimpleFlight reports cutting tracking error by over 50% versus prior methods on a single quadrotor. For multi-drone payload transport under sim-to-real shift, there is no reported result found on Lacuna, so this baseline's numbers will be produced by us. It will use identical simulator samples, network sizes, environment steps, and hyperparameter search budget.

The key ablations isolate the ingredient. The first is full-team belief sharing ($\kappa = N$), which tests whether locality costs anything. The second is recurrent MAPPO with a learned communication channel of the same bandwidth but no supervised type targets, which tests whether explicit discrepancy inference matters or whether hidden states learn it anyway.

A robust-control comparator will use worst-case-over-bracket training in the spirit of Sample-Efficient Distributionally Robust MARL. The exact algorithm only runs on tiny games, so we instantiate a tractable version that trains each drone against adversarial neighbor types. Its purpose is to quantify conservatism, not to be beaten on guarantees.

Winning means lower payload tracking error and higher success than the self-inference baseline on coupled tasks, with no loss on nominal hardware. It also means completion times close to an oracle that is given true parameters.

Why not just…?

Why not just tune medium domain randomization per team size? This is cheap and worked at $N=3$. But the tuned width is a single global dial. It cannot express that robot 2 is weak while robot 5 is fine, and the Duckietown authors themselves note there is no principled way to set it. Our width-versus-$N$ sweep will show whether a well-tuned dial closes the gap. If tuned randomization matches our method at every $N$, we will report that the inference machinery is unnecessary.

Why not just let each robot identify its own dynamics? This is the real threat. On loosely coupled tasks we expect it to match us, and we will say so. The question is settled by a coupling sweep: the same drones in formation flight, with weak coupling, versus rigid multi-drone payload transport, with strong coupling. The idea stands only if neighbor beliefs reduce payload error specifically where coupling is strong. If self-only inference ties everywhere, the contribution reduces to a negative result about the need for joint reasoning, which is still worth knowing.

Why not just use recurrent policies with learned communication and let them infer teammates implicitly? They might. The matched-bandwidth, unsupervised-channel ablation tests this directly. BATPAL's evidence that an explicit belief over severity brackets avoids both conservatism and over-optimism suggests explicit supervision helps, but that evidence comes from adversaries, not physics. If the implicit channel matches, the supervised type head is dropped.

Why not independent learners with the single-agent recipe? Independent learners' error grows with population in Learning Mean-Field Games. Independent Q-learning trails centralized-critic methods in The StarCraft Multi-Agent Challenge. The robust-games analysis predicts that mismatch cascades between teammates. We include independent PPO with SimpleFlight's recipe as a lower reference rather than a main competitor.

Proposed Experiments

Simulation experiments scale formation flight and suspended-payload transport in OmniDrones over $N = 2, 4, 8, 16$. We evaluate on a held-out "reality proxy" simulator that adds effects absent from training: an inter-drone downwash force model, extra actuator lag, sensor noise, and communication delay. Metrics are task success, payload position error, completion time relative to the parameter-aware oracle (our conservatism measure), action jerk, and bytes communicated per step.

The decisive checks are three. First, a coupling-by-$\kappa$ interaction: neighbor inference should beat self-inference in transport and not in formation flight. Second, a flat $\kappa=1$ versus $\kappa=N$ curve, whose cost diverges with $N$. Third, conservatism under fixed randomization that worsens with $N$ while ours stays flat.

Hardware experiments first replicate the three-robot Duckietown protocol with deliberately mismatched wheel trim across robots, compared against the 1.85× reference. They then run 2–4 Crazyflie drones carrying a tethered payload with mismatched added masses, using SimpleFlight's transfer pipeline. As in prior work, we rely on motion capture for state. We state this as a limitation: the study isolates dynamics discrepancy, not perception.