When a swarm of robots is taught by example, the examples are rarely perfect. Exact multi-robot planners become intractable beyond a few dozen agents. The cheaper planners and human operators that replace them produce trajectories that are mostly sensible but contain local failures: two robots deadlocked in a corridor, one robot parked on its goal while blocking everyone else, a needless detour. Current methods handle this badly. The large-swarm systems assume a perfect planner and patch the remaining failures with hand-written penalty terms. The imitation methods that tolerate imperfect experts score whole trajectories and were built for a single agent. We propose to score demonstration quality at the unit where swarm mistakes actually happen: one agent, its nearest neighbors, and a short window of time. A discriminator, meaning a classifier that tells "good" behavior from "other" behavior, is trained on these local windows. It is permutation-invariant (it does not care how neighbors are ordered or how many there are), so the same scorer works at any population size.
This single object does three jobs. It separates the good parts of a flawed trajectory from the bad parts, which assigns blame among agents without any labels. It provides a dense reward that replaces hand-written penalties such as a "blocking penalty." Because it only ever looks at a bounded neighborhood, it can serve as a training reward at swarm sizes for which no demonstrations exist at all. The practical result would be a pipeline that trains on small, messy demonstration sets and deploys zero-shot to hundreds of agents, with no hand-tuned reward terms.
Core Claim
The claim is that in decentralized swarms, demonstration quality should be estimated per local interaction window, not per trajectory or per episode. A trajectory-level filter throws away an entire 64-agent episode because one pair deadlocked, or keeps it and imitates the deadlock. A window-level scorer keeps the 62 good agents' behavior and down-weights the two that failed.
The second claim is that the local scorer, trained on small populations, remains a valid reward when applied to larger ones. This lets adversarial imitation continue at scales beyond the demonstrations, instead of only transferring a frozen policy.
Both claims are falsifiable. If trajectory-level filtering matches window-level filtering, the first claim fails. If the discriminator's reward stops tracking true task quality as density rises, the second claim fails.
Where This Sits
Three strands of prior work each supply part of the picture.
Large-swarm learning. Zero-shot swarm work shows that policies built on local, order-free inputs transfer from few agents to many:
- PRIMAL combines reinforcement learning with cloning an optimal centralized planner and scales pathfinding to 1,024 agents. It needs a hand-tuned blocking penalty to avoid corridor gridlock.
- GNN path planning goes from 10 to 60 robots using Conflict-Based Search corrections queried online. This assumes an optimal expert is always available.
- Decentralized barrier certificates transfer from 8 to 1,024 drones using RowMax pooling. RowMax is a max over neighbors that ignores their order and count. That work notes compounding neighbor drift in closed loop.
These papers supply our architecture: local observations and order-free pooling. They motivate the proposal because each relies on either an optimal expert or engineered reward terms.
Imitation from imperfect experts. This strand supplies the scoring machinery:
- DWBC trains a positive-unlabeled discriminator to weight suboptimal offline data. Positive-unlabeled means a classifier learns from a small set of known-good examples and a large unlabeled pool.
- PN-GAIL penalizes imitation of known-bad transitions using a 20% labeled seed.
- SAIL self-paces which demonstrations the critic trusts.
- RelaxDICE relaxes exact distribution matching to the data support.
All four are single-agent, so their unit of quality is a transition or trajectory of one body. None addresses the case where a joint trajectory fails because of one agent.
Multi-agent imitation. Two papers constrain the design:
- MAGAIL shows adversarial imitation beats cloning in multi-agent particle worlds. Its centralized critic limits it to 2–4 agents, and it assumes a near-optimal expert.
- MisoDICE is the closest work: offline multi-agent imitation from mixed-quality data. It ranks whole trajectories with GPT-4o and works at fixed team sizes, which does not carry over to raw geometric swarm states or changing population.
Imitation in mean-field games proves that adversarial imitation errors grow exponentially with horizon when dynamics depend on the population distribution. This is a warning that a discriminator blind to neighbors will fail at scale. Our discriminator conditions on exactly the neighborhood that couples the dynamics. That is similar in spirit to their mean-field-control fix, but evaluated with imperfect experts and real swarm tasks rather than toy attractor games.
Research Program
The discriminator. The scored object is a window $x_i^t$: agent $i$'s own state change over $w$ steps, together with the relative positions and velocities of its $k$ nearest neighbors over the same steps. Following observation-only imitation, windows use state changes rather than actions. This lets demonstrations come from planners, logged robots or teleoperation alike. The discriminator $D(x_i^t)$ pools neighbors with attention or RowMax. Attention's convergence to a mean-field limit is argued by Biswal.
We fix $k$ rather than taking all neighbors within a radius. Density at large scale then changes which neighbors are seen, but not how many. This follows the pruning logic of Fu et al..
Training the discriminator. It is trained positive-unlabeled, as in DWBC:
- The positives are windows from a small clean seed: exact Conflict-Based Search solutions at 4–8 agents, where exact solving is cheap.
- The unlabeled pool is windows from a large suboptimal corpus: prioritized planning, ORCA-style reactive avoidance (which the PRIMAL authors report deadlocking beyond 16 agents in clutter), and injected operator-style errors.
The window-level weight $\omega_i^t = D(x_i^t)/(1-D(x_i^t))$ enters two stages.
Stage one: offline weighted cloning. The objective is
$$\mathcal{L}(\theta) = -\sum_{i,t} \omega_i^t \log \pi_\theta(a_i^t \mid o_i^t),$$
where $\pi_\theta$ is the shared decentralized policy and $o_i^t$ its local observation. When a demonstration lacks actions, an inverse model trained on simulator rollouts labels them.
Stage two: online adversarial fine-tuning. Each agent receives the reward $r_i^t = \log D(x_i^t) - \log(1 - D(x_i^t))$. A sparse team-success indicator may be added, and we ablate it; no shaped terms are allowed. The discriminator keeps updating against policy rollouts, with the clean seed remaining the positive class.
This online stage is run on a population curriculum: 8, then 32, then 128 agents. The larger sizes have no demonstrations. The reward is valid there only if the local scorer transfers, which is claim two.
Analyses. We will measure:
- whether the scorer localizes injected errors to the right agent and time;
- whether its reward at large $N$ correlates with ground-truth local quality, measured by each agent's blocking events and its marginal contribution to makespan (the time until the last agent reaches its goal);
- whether goal-blocking is penalized without being written down. We check this by computing the reward on scripted "park on goal in a corridor" windows.
Baselines
Naive reference points. The first is plain behavior cloning on the full suboptimal corpus. The second is the same cloning on only the clean seed. Both use the identical policy network, observation, and gradient-step budget as our method. Beating them establishes that there is signal in the imperfect data at all.
The strongest existing solution on the pathfinding question. This is PRIMAL: reinforcement learning plus cloning, with its hand-tuned blocking penalty. The PRIMAL authors report a "near-perfect success rate for up to 1,024 agents" in low obstacle density gridworlds. They report no numeric value in our file for high density. We run it with its published reward terms, given the same environment steps as our method. It is allowed demonstrations from an exact planner where tractable, which is an advantage over us. Our method counts as beating it if it matches PRIMAL's success rate at low density and exceeds it at 20–30% obstacle density. It must do this without the blocking penalty and with only suboptimal demonstrations beyond the 8-agent seed.
An oracle reference. GNN planning with online Conflict-Based Search corrections serves as an upper-reference oracle, run with its exact expert queries. No reported success-rate number was found on Lacuna for its 10-to-60 transfer; the file only states it generalized without fine-tuning.
Direct competitors on imperfect-demo multi-agent imitation. Three methods apply:
- MAGAIL with decentralized discriminators, run at its native scale and with our local encoder.
- DWBC applied per agent, scoring whole single-agent trajectories.
- MisoDICE on SMACv2.
On Protoss 5_vs_5 with the mixed split of 200 expert and 1,000 poor trajectories, MisoDICE reports a 20.7% win rate. It reports 14.2% for expert-only cloning and 12.4% for independent DemoDICE. We use the same split and hold our tuning budget equal to the budget we give re-run baselines. Beating MisoDICE means exceeding 20.7% without any language-model ranking.
Ablations isolating the key ingredient. The central ablation keeps everything fixed but changes the scoring unit from window to whole agent-trajectory to whole episode. A second ablation removes neighbor input from the discriminator, making it ego-only. A third varies the clean seed from 0% (self-paced thresholding in the style of SAIL) to 20%.
Why not just…?
Why not just filter whole trajectories by planner cost and clone the rest? This is the most likely threat. If failures in the corpus are rare and clustered in whole episodes, it could be as good as our method. The injected-error experiment settles it directly. We vary how many agents per episode fail, from one to most. Our prediction is that trajectory filtering is competitive only when failures are episode-wide, and loses when failures are sparse and local. That is the regime heuristic planners produce. If it wins across the board, the core claim is wrong and we will report that.
Why not just use weighted offline cloning and skip the adversarial loop? DWBC shows offline weighting can beat RL-based offline imitation on 27 of 36 D4RL tasks at about 7x speed. So stage one alone might suffice at training scale. Stage two is justified only if closed-loop drift appears at large $N$, which Qin et al. flag and MAGAIL's advantage over cloning suggests. The stage-one-only row at 512 and 1,024 agents answers this. If it matches the full method, we drop the min-max game.
Why not just train with online multi-agent RL on sparse success plus distance and collision terms? That is precisely the manual reward engineering we want to remove. Daniel et al. document proxy-reward hacking. PRIMAL itself needed a blocking penalty to avoid gridlock. We include this baseline with an equal environment-step budget and count corridor deadlocks explicitly.
Why not just generate exact-planner demonstrations at small scale and clone them, as the GNN work does? Where an exact expert is cheap, that approach is sound. It is our oracle row. The proposal targets the settings where it is not available: dense maps, continuous dynamics, and human or heuristic sources. There, only a small exact seed is affordable.
Proposed Experiments
Pathfinding gridworlds. The main testbed is the gridworlds used by PRIMAL and the GNN planner: 20×20 to 160×160 maps at 0–30% obstacle density. Demonstrations use 8 agents, and deployment goes up to 1,024. We report:
- success rate;
- flowtime (the sum of all agents' travel times);
- makespan;
- number of goal-blocking events.
Continuous navigation. The second testbed is the Navigation and 3D Maze tasks from the barrier-certificate benchmark, with safety rate added. There MDBC reports near 99% at 32 agents versus about 92% for MADDPG-Safe; these are reference points, not our direct competitors.
StarCraft. The third testbed is SMACv2, trained at 5v5 and evaluated at 5v5 for the MisoDICE comparison. We also evaluate zero-shot at 10v10, where no prior number exists.
Decisive checks. Three results decide the proposal:
- Error localization. Injected per-agent errors must be recovered with clearly higher accuracy by window scoring than by trajectory scoring.
- Reward transfer. Discriminator reward must stay correlated with ground-truth local quality from 8 to 512 agents, including at the densest maps.
- Replacing hand-written penalties. Without a blocking penalty, blocking events must fall to or below PRIMAL's level.
A failure on the second check at high density, with success at low density, would locate the limit of zero-shot reward transfer. That would itself be a useful result.
References
- PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning
- Graph Neural Networks for Decentralized Multi-Robot Path Planning
- Learning Safe Multi-agent Control with Decentralized Neural Barrier Certificates
- Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations
- PN-GAIL: Leveraging Non-optimal Information from Imperfect Demonstrations
- Robust Adversarial Imitation Learning via Adaptively-Selected Demonstrations
- Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
- Multi-Agent Generative Adversarial Imitation Learning
- MisoDICE: Multi-Agent Imitation from Mixed-Quality Demonstrations
- On Imitation in Mean-field Games
- Imitation Learning from Observation
- Universal Approximation of Mean-Field Models via Transformers
- Concentration Network for Reinforcement Learning of Large-Scale Multi-Agent Systems
- Active Reward Learning