One person cannot teleoperate four robot arms at once. This is one of the most concrete reasons multi-robot imitation learning lacks data. The workaround is to demonstrate for one robot at a time while the others run their current, imperfect learned policies. A recent study showed this works surprisingly well, but it rotates control on a fixed schedule and spends human effort uniformly across robots and situations. This note proposes a different collection procedure. At each step the learner decides which robot the human should take over and in which situation, aiming at the robot whose policy is most uncertain given how its teammates currently behave. The goal is a team policy for $N$ coupled robots whose demand for human demonstrations grows roughly with the sum of the robots' individual difficulties rather than with the size of their joint action space.
There is also a theoretical reason to expect this to work. Offline multi-agent learning theory has shown that a dataset only needs to cover unilateral deviations: one agent trying different actions while the others follow fixed strategies. This weaker condition makes sample requirements add up across agents instead of multiplying. Round-robin teleoperation produces exactly this kind of data, since one agent is driven by the human while the rest follow their policies. Choosing what to collect, rather than collecting on a fixed rotation, turns this coincidence into a data-collection algorithm that can be analyzed, built, and tested.
Core Claim
Per-agent, uncertainty-directed round-robin querying can reach a stable coordinated team policy with substantially fewer human-labeled steps than fixed-rotation round-robin collection on tasks with three or more physically coupled robots. A stable team policy here is one where no single robot can improve by changing its own behavior. The gap should widen as the number of robots grows. In tabular and linear settings, the number of queries should scale with $\sum_i |A_i|$ (or $\sum_i d_i$ for feature dimensions) times a per-agent coverage term, never with $\prod_i |A_i|$. Here $|A_i|$ is agent $i$'s action count and $d_i$ its feature dimension. The part that is new is the selection rule over the pair (which robot, which state), conditioned on teammates' current policies. Round-robin data collection already exists, as do uncertainty-driven expert queries and additive offline bounds.
How This Relates to What Exists
Three papers motivate the idea and supply its pieces. R2BC introduced round-robin behavior cloning: the human controls one robot while the others act autonomously, and teammates' mistakes naturally elicit recovery demonstrations. On physical mobile robots it beat centralized cloning by 3.25x on navigation and 5.9x on block pushing. It uses a fixed rotation, was tested only with two to three agents, and gives no bounds. We keep its data-collection interface and replace the rotation with a learned query rule.
Provably Efficient Offline MARL via Strategy-wise Bonus proves that unilateral coverage yields sample complexity additive in $\sum_i A_i$. It assumes the dataset is given and does not ask how to obtain it. We supply the collection side.
Learning Equilibria from Data proves that static behavior cloning suffers linear regret when demonstrations miss deviation states. Its interactive algorithm MURMAIL queries the expert at maximally uncertain states and reaches $O(\epsilon^{-8})$ sample complexity. That result is limited to two-player zero-sum tabular games and assumes an expert that labels the game's joint play. Our setting is cooperative, has $N$ agents, and allows only one robot to be labeled per query, which is the constraint teleoperation actually imposes. Freihaut et al.'s negative result also constrains us: it predicts that fixed-rotation collection will plateau whenever rotation fails to visit the states where a teammate's errors push the system.
On the policy side, we reuse rather than invent. The decoupled interaction framework gives one policy per arm plus a learned module that injects the other arm's features. It reported a 23.5% average-success gain over the 3D Diffusion Policy baseline on RoboTwin, with one-sixth the parameters. That per-arm structure is what our per-agent uncertainty estimates attach to.
CLAS attacks the joint action explosion by compressing actions through a learned latent space, trained with reinforcement learning and dense reward. It was the only method to solve four-arm lifting in robosuite. We differ in the data regime: we assume human demonstrations under a strict budget rather than hundreds of thousands of reward-labeled environment steps.
Low-rank and linear value factorizations, namely Tesseract and DOP, compress the joint value function but presuppose reward-driven reinforcement learning. They are orthogonal to our question of how to spend human labels.
One known limitation constrains the claim. Round-robin data, like unilateral coverage, can only certify that no single robot benefits from deviating. In a cooperative task with several incompatible coordination conventions, it cannot move the team from a worse convention to a better one. We treat this as a measured failure mode, not something to hide.
Research Program
The algorithm maintains an ensemble of per-arm policies, each conditioned on its own observation and on teammate features through the decoupled framework's interaction module. During collection, all robots run their current policies. At each decision point the learner scores every pair (robot $i$, current state $s$) by ensemble disagreement on robot $i$'s action, a stand-in for how unsure the policy is. The score is weighted by how much $s$ lies on the distribution induced by teammates' current policies. When the top score crosses a threshold, control of robot $i$ is handed to the human for a short segment, and the labeled segment is added to robot $i$'s buffer.
This is the "others-integrated" view of uncertainty that robust MARL work used to make uncertainty sets additive across agents. We apply it to the learner's epistemic uncertainty rather than to environment dynamics.
The theoretical part targets a cooperative Markov game with independent linear features, following the setting of Cui, Zhang and Du. The aim is to show that the number of single-agent queries needed for an $\epsilon$-unilaterally-stable team policy scales polynomially with $\sum_i d_i$, the horizon, and $1/\epsilon$. The per-agent coverage term would be measured under teammates' policies rather than the all-deviation coefficient. We hope for an exponent well below MURMAIL's eighth power, because labels are requested for one agent's local action rather than for an equilibrium of the full game. We do not promise this. A matching lower bound for fixed-rotation collection, showing it can waste queries on agents that are already accurate, would make the separation sharp.
The empirical part uses a scripted or reinforcement-learning expert as a simulated teleoperator, so that query counts are exact and repeatable. We then run a small human study on hardware to check whether takeover segments chosen by the algorithm are tolerable for a person.
Baselines
The naive reference point is joint behavior cloning on synchronized demonstrations from the simulated expert, given the same number of labeled robot-steps. A labeled robot-step is one robot's action at one timestep. Under this accounting, a synchronized $N$-robot demonstration costs $N$ times as much as a single-robot step. Joint cloning is unrealistic for humans when $N \ge 3$, but in simulation it shows what synchronized data would buy. Beating it at equal labeled robot-steps would show that targeting matters, not just that round-robin is cheaper to collect.
The strongest competitor is R2BC with its fixed rotation. It is run with the identical per-arm policy architecture, the identical simulated expert, the same total budget of labeled robot-steps, and the same hyperparameter search budget. The only difference is how takeovers are scheduled. The file reports R2BC matching or exceeding joint cloning in all four VMAS domains and the hardware ratios above. No numerical VMAS success rate for R2BC was found on Lacuna, so we will reproduce it ourselves. Beating it means reaching R2BC's final success rate with at most half its labeled robot-steps on three-robot tasks, with the advantage growing at four robots.
A reward-driven reference is CLAS on robosuite four-arm lifting. The file reports it as the only method to solve the task, with peak dual-arm peg-in-hole performance at roughly 250,000 steps. We compare at equal wall-clock robot interaction time to ask whether a small human budget substitutes for that reinforcement-learning interaction. No query-count result exists for CLAS because it uses none.
Two ablations isolate the key ingredient. One keeps uncertainty-driven state selection but picks the robot at random. The other picks the robot by uncertainty but triggers takeovers at random times. If either matches the full method, the joint (robot, state) selection is not what matters.
Why not just…?
Why not just run R2BC longer? Fixed rotation does converge eventually. The question is cost, and Freihaut et al.'s linear-regret result predicts that uniform collection stalls when rare teammate-induced states go unvisited. If plain R2BC matches our curves at equal budget, the proposal fails on its central claim. The equal-budget comparison is designed to show exactly that.
Why not just pretrain on synthetic data or a big model and fine-tune? RoboTwin 2.0 reports a 367% relative gain over a 10-demonstration real-only baseline. TwinVLA reaches 71.0% real-robot success from twin single-arm backbones. Both are real threats for two arms. Both, however, are fixed to dual-arm layouts, and RoboTwin reports under 15% success on tightly synchronized contact tasks such as Dual Shoes Place. Pretraining and query selection compose rather than compete, so we will run our method on top of a pretrained per-arm initialization and check whether the query savings survive.
Why not just use a two-arm teleoperation rig? For $N=2$ that is the right answer, and we concede the dual-arm case. The proposal is therefore evaluated mainly at three and four robots, where no single-operator rig exists. Two-arm results serve only as a sanity check.
Why not just query by joint-policy uncertainty, DAgger-style? A joint query requires a synchronized expert, which is the very thing that does not exist. Collapsing joint uncertainty into one scalar also discards which robot is responsible. The robot-random ablation measures how much that attribution is worth.
Proposed Experiments
The main testbed is VMAS Balance, Transport, and Buzz Wire from R2BC, plus robosuite dual-arm lifting and four-arm lifting, extended to four arms where needed. We measure success rate against cumulative labeled robot-steps, and how the number of labeled robot-steps needed to reach 80% of expert success scales with the number of robots. On a tabular cooperative game with known structure, we also measure the gap from unilateral stability.
The first decisive check concerns scaling across two, three, and four robots. If query cost grows roughly linearly in $N$ for our method and faster for fixed rotation, the additive claim holds empirically.
The second decisive check is a two-convention task, for example lifting from either side, where a better convention exists. We report how often each method gets stuck in the worse convention. This directly measures the limitation inherited from unilateral coverage.
A final hardware study uses three robots and a human operator. It reports success, query count, and the operator's self-reported workload, the cost R2BC left uncharacterized.
References
- R2BC: Multi-Agent Imitation Learning from Single-Agent Demonstrations
- Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonus
- Learning Equilibria from Data: Provably Efficient Multi-Agent Imitation Learning
- Rethinking Bimanual Robotic Manipulation: Learning with Decoupled Interaction Framework
- CLAS: Coordinating Multi-Robot Manipulation with Central Latent Action Spaces
- Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning
- DOP: Off-Policy Multi-Agent Decomposed Policy Gradients
- Breaking the Curse of Multiagency in Robust Multi-Agent Reinforcement Learning
- Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximation
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation
- TwinVLA: Data-Efficient Bimanual Manipulation with Twin Single-Arm Vision-Language-Action Models
- RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning