Imitating Robot Fleets From Small-Fleet Demonstrations

When many identical robots share a space, such as delivery rovers on a sidewalk, warehouse carts in an aisle, or a swarm covering targets, each robot's motion depends on how crowded its surroundings are. Demonstrating good fleet behavior is expensive. A human can teleoperate one robot, or perhaps coordinate four, but nobody will demonstrate fifty robots at once. The natural shortcut is to clone the behavior of a few demonstrated robots and copy that policy onto every robot in a larger fleet. That shortcut has a known failure. When the crowd's distribution changes the physics each robot experiences, a small imitation error in every robot shifts the crowd, the shifted crowd changes everyone's dynamics, and the error feeds back on itself. On Imitation in Mean-field Games proves that in this setting both behavior cloning and standard adversarial imitation have error that grows exponentially in the task horizon. It also proves that an imitation method whose inner loop solves a mean-field control problem keeps the growth quadratic. Mean-field control means optimizing a single shared policy while accounting for how that policy reshapes the whole population's distribution.

That result has only been shown on a tiny synthetic discrete game. This note proposes to make it a usable robot-learning method and to test whether it matters in practice. The method learns a continuous-action fleet policy from demonstrations collected with only a handful of robots, then deploys it at much larger fleet sizes. Two ingredients make this possible. The first is a permutation-invariant summary of the local crowd, built from kernel mean embeddings, so the same policy runs unchanged at any fleet size. The second is a mean-field-control inner loop, so training accounts for how the learner's own errors move the crowd. The concrete claim is falsifiable. As congestion coupling increases, cloning-based fleet policies should degrade exponentially with horizon, while the mean-field imitation learner should not. That advantage should survive the jump from four demonstrated robots to a hundred deployed ones.

Core Claim

The data-scarcity problem in fleet robotics is not mainly a shortage of single-robot trajectories. It is a shortage of demonstrations at the crowd densities the fleet will actually create. Mean-field theory supplies two things. First, a policy defined on the robot's own state and an embedding of the surrounding distribution is the same object at any fleet size $N$, so small-fleet demonstrations are in principle informative about large fleets. Pessimism Meets Invariance supports this with an offline bound independent of agent count and with empirical stability from 15 to 100 agents. Second, the Ramponi et al. dichotomy predicts exactly when naive cloning will fail: when the crowd distribution $\mu$ enters the transition kernel $P(s' \mid s, a, \mu)$. Their bound for behavior cloning in that regime scales as

$$\text{imitation gap} = \mathcal{O}(L_P^{H}),$$

where $L_P$ is the Lipschitz constant of the dynamics with respect to $\mu$ (how strongly crowding changes motion) and $H$ is the horizon. What is new here is not the mean-field imitation idea, which belongs to Ramponi et al. The new parts are three. We extend it to continuous states and actions with empirically estimated densities, which are two of the open problems flagged in Scalable Deep Reinforcement Learning Algorithms for Mean Field Games. We apply it to the specific robotics problem of demonstrating at small $N$ and deploying at large $N$. And we run a direct empirical test of whether the predicted exponential-versus-quadratic split appears when $L_P$ is dialed up in a robot-like setting.

Research Program

The method has three parts.

The first is the policy representation. Each robot observes its own pose, velocity and goal, plus a kernel mean embedding of neighbor positions and velocities within a sensing radius. A kernel mean embedding is a fixed-length vector that averages a feature map over a set of points, so it does not depend on how many points there are or their order. We reuse this from SAFARI (Pessimism Meets Invariance), which used mean embeddings for offline value learning with known rewards. We use them for reward-free imitation and as policy input. The action head is a Gaussian policy for the adversarial learner. A diffusion-policy head is a secondary variant, so we can check whether expressive action models change the picture.

The second is the imitation objective. A discriminator distinguishes expert (state, crowd-embedding, action) tuples from learner tuples and defines a reward, as in adversarial imitation. The distinguishing step is the inner loop. In the "Vanilla-ADV" analogue, one learner robot optimizes this reward against a crowd flow frozen from the previous iteration. That is a best response to a fixed crowd, which is the Nash-style loop Ramponi et al. show compounds exponentially. In our mean-field-control loop, all $N_{\text{train}}$ simulated robots share one policy, and we maximize the population-averaged discriminator reward with on-policy gradients through full particle rollouts. The crowd therefore moves as the policy changes, and the gradient sees that movement. This sidesteps the discrete-action softmax bottleneck of Munchausen mirror descent (Laurière et al.), because we never need a mixture of past policies. Mean-field control seeks a social optimum rather than a Nash equilibrium, so it needs only one policy.

The third is the demonstration regime. Experts are available for only $N_{\text{demo}} \in \{2, 4, 8\}$ robots and only a small number of trajectories, from 10 to 200. Training rollouts in the inner loop may use larger simulated $N_{\text{train}}$, because simulation is cheap and demonstrations are not. Evaluation is at $N_{\text{deploy}}$ up to 100.

The analysis component measures the imitation gap as a function of $H$ and of the congestion coupling. We fit the slope of log-error against $H$ and check whether it tracks $\log L_P$ for the cloning baselines, as the theory predicts.

Baselines

The naive reference point is plain behavior cloning on the small-fleet demonstrations, with the policy deployed unchanged on every robot of the large fleet. It uses the same network, the same crowd embedding as input, and the same demonstrations. It is the realistic default practice, and it is the method Ramponi et al. predict will blow up. To be fair, we tune its learning rate, depth and regularization on a held-out validation split with the same hyperparameter budget given to our method.

The strongest practical competitor is behavior cloning with crowd-density augmentation and noise injection. During cloning we perturb demonstrated states and synthetically resample neighbor sets at higher densities, so the embedding sees out-of-distribution crowds. This is the reviewer's "just augment the data" answer, in the spirit of the coverage-expansion approaches that work well for single-robot manipulation, such as MimicGen. It addresses distribution shift without any inner loop, so it is the most likely to match us. It receives the same simulator access and wall-clock budget as our method's inner loop.

The closest method baseline is Vanilla-ADV: adversarial imitation with the frozen-crowd, best-response inner loop described above. It shares our discriminator, policy, and inner-loop simulator budget, differing only in whether the crowd is held fixed during policy optimization. This doubles as the key ablation, because it isolates the single ingredient the theory says matters.

On a discretized grid version of the congestion task, we also run MF-AIRL from Adversarial Inverse Reinforcement Learning for Mean Field Games. It recovers rewards in discrete mean-field games and shows faster convergence than earlier mean-field IRL methods, but it is limited to finite discrete spaces. Including it tells us how much is lost or gained by moving to continuous control.

Two references bound the problem. The first is SAFARI (Chen et al.), given privileged access to rewards, as an offline-RL upper reference. The second is a DAgger-style oracle that queries the expert at deployment $N$, which is exactly the resource we assume is unavailable.

No reported result was found on Lacuna for imitation learning of continuous-control robot fleets under population-dependent dynamics. The closest reported evidence is qualitative. SAFARI "performed comparably to COMA," and its gap to baselines "did not widen" from 15 to 100 agents on cooperative navigation. MFC-ADV stabilizes error on the synthetic Attractor game. Beating the baselines therefore means being judged against our own controlled runs. At the high-coupling setting and $N_{\text{deploy}}=100$, the mean-field learner must have a lower measured imitation gap and higher task success than augmented cloning across seeds. Its log-error-versus-horizon slope must also be significantly flatter than both cloning variants.

Why not just…?

Why not just augment the cloning data with denser synthetic crowds? This is the real threat, and the proposal treats it as one. Augmentation enlarges the set of crowd states the policy has seen. It does not fix the feedback loop in which the learner's own errors produce crowds the expert never would. Ramponi et al.'s exponential bound concerns that loop, not missing coverage. Our experiment settles the question directly. If augmented cloning matches the mean-field learner's horizon slope at high coupling, we will report that the theory's failure mode is practically fixable by augmentation, and that would itself be a useful result.

Why not just train the fleet with multi-agent reinforcement learning on a hand-written reward? Fleet behavior that people want, such as courteous yielding, lane formation, or a particular spacing style, is precisely what is hard to write as a reward. That is why demonstrations are collected at all. Where a reward is known, we include SAFARI with privileged rewards as an upper reference, so the cost of not knowing the reward is measured rather than assumed.

Why not just collect demonstrations at the deployment fleet size? For humans demonstrating, this is infeasible beyond a few robots. In simulation, if a scalable expert already existed there would be nothing to imitate. Our DAgger oracle quantifies how much large-$N$ expert access would be worth, which bounds the value of the problem we are solving.

Why not just use single-robot model-based relabeling, as LogoNav does? Learning to Drive Anywhere with Model-Based Reannotation relabels noisy and passive navigation data with a single-robot forward model. It reports 0.857 long-horizon goal success versus 0.333 for NoMaD, but on single-robot navigation, not fleets. Its forward model has no crowd term, so its relabeled actions are correct only at the crowd densities seen in the data. It is complementary as a front-end for cleaning demonstrations, not a substitute for accounting for the learner's effect on the crowd.

Proposed Experiments

The testbeds are drawn from the file. The first is Multi-Agent Particle Environment cooperative navigation as used by Chen et al., with $N$ from 15 to 100. The second is the 2D crowd-with-congestion model from Laurière et al., extended to continuous unicycle kinematics. We modify both so that each robot's achievable speed falls with local density through a coupling coefficient. This gives a controlled $L_P$ that we sweep from zero, where the crowd affects only rewards and Ramponi et al. predict no exponential blow-up, to strong.

Experts come from solving the continuous mean-field control problem with the direct-control solver of Deep Learning for Mean Field Optimal Transport, which reports 1.2–5.6% relative total-cost error on problems with analytic solutions. Having a known expert lets us compute the true imitation gap rather than a proxy.

Metrics are three. The first is the imitation gap: the learner's cost minus the expert's, at deployment $N$. The second is task success, meaning landmark coverage or goal arrival within the horizon. The third is throughput and collision rate. We sweep the horizon $H$, the coupling strength, $N_{\text{demo}}$, the number of demonstration trajectories, and $N_{\text{deploy}}$, with five seeds each.

There are three decisive checks. The first tests the theory. At zero coupling, all methods should have similar horizon slopes. As coupling grows, the cloning baselines' log-gap slope should rise roughly with $\log L_P$, while the mean-field learner's should stay near the quadratic regime. The second is the ablation. Freezing the crowd in the inner loop (Vanilla-ADV) should recover the cloning-like slope. If it does not, the mean-field-control ingredient is not what is helping. The third tests the data-scarcity payoff. A policy trained from four-robot demonstrations should retain most of its performance at 100 robots, and this retention should exceed augmented cloning's. If augmented cloning matches on all three checks, the honest conclusion is that mean-field structure is useful as an input representation but not as a training loop, and we will say so.