Imitation Bootstrapped Multi-Agent Reinforcement Learning

Tony H. Zhang, Connor Mattson, Daniel S. BrownarXiv 2026 · arXiv

TL;DR

Imitation Bootstrapped Multi-Agent Reinforcement Learning (IBMARL) is a framework that significantly accelerates multi-agent training by "bootstrapping" from easy-to-collect single-agent demonstrations. By using a centralized critic to intelligently arbitrate between imitation and RL action proposals, IBMARL produces decentralized agents that learn up to 4.2x faster than existing baselines and can even surpass the performance of the experts who taught them.

What Problem Does This Paper Address?

Multi-agent reinforcement learning (MARL) is notoriously difficult because the "joint-action space" (the total number of possible combinations of actions for all agents) grows exponentially with every agent added. Furthermore, in cooperative tasks with sparse rewards, agents can wander aimlessly for millions of steps before ever stumbling upon a successful strategy.

While human demonstrations can guide exploration in single-agent RL, they are hard to apply to MARL. It is nearly impossible for one person to teleoperate a whole team of robots simultaneously to provide "joint" demonstrations. Existing workarounds, like Round-Robin Behavior Cloning (R2BC), allow a human to train agents one by one, but the resulting imitation learning (IL) policies are often mediocre and capped by the human's own skill level.

The authors of IBMARL set out to bridge this gap: How can we use these imperfect, single-agent demonstrations to jumpstart MARL without being limited by the demonstrator’s performance?

Methods

IBMARL extends a single-agent technique called Imitation Bootstrapped RL (IBRL) to the multi-agent domain. The process follows a "Teach-Act-Learn" cycle:

  1. Teach: A human uses R2BC to control one robot at a time while the others run their current policies. This creates a set of "frozen" IL policies for each agent.
  2. Act (The Arbiter): During training, every agent generates two action proposals: one from its IL policy and one from its evolving RL policy. A centralized critic ensemble (the "arbiter") evaluates all $2^N$ possible combinations of these actions and selects the best one for the team to execute.
  3. Learn (Gated Imitation): The experience is stored in a replay buffer to train the critics. Crucially, the authors introduce a gated imitation penalty. This is a soft "Q-filter" that pulls the RL policy toward the IL action only in states where the critic believes the IL action is actually better.
Figure 2: IBMARL Pipeline. Left: R2BC trains the decentralized IL policy 𝜇𝜓from single-agent demonstrations (Sec. 3.2). Middle: the agents’ IL and RL proposals form the 2𝑁joint candidates A(o), which the target critics score and the arbiter samples (Eqs. (5)–(6)). Right: the critics bootstrap through the same arbiter (Eq. (7)), and the actor loss adds the gated imitation term (Eqs. (8)–(9))

At deployment, the centralized critics and IL policies are discarded; each agent runs its own decentralized RL policy.

Datasets

The researchers evaluated IBMARL using the Vectorized Multi-Agent Simulator (VMAS) on four cooperative tasks:

  • Navigation: Three agents must reach separate goals.
  • Buzzwire: Two agents must carry a ball through a narrow corridor without touching the walls.
  • Balance: Three agents must lift a beam to move a package to a goal.
  • Transport: Three agents must collaborate to push a heavy package.

They used both synthetic experts (pre-trained RL agents) and real human demonstrations (collected via an Xbox gamepad) to provide the initial IL policies.

Key Insights and Ideas

  • Imitation as a Participant, Not Just Data: Most "RL from demonstrations" methods just dump expert data into a replay buffer. IBMARL keeps the IL policy "in the loop" as an active participant that proposes actions during exploration.
  • Per-Agent Mixing: The arbiter doesn't just choose between "all IL" or "all RL." It can pick IL for Agent A and RL for Agent B. This allows the team to improve incrementally, agent-by-agent, which is much easier than trying to improve the entire team's joint strategy at once.
  • Breaking the Performance Ceiling: Because the gated penalty only applies when the IL policy is superior, the RL agents are free to diverge from the demonstrations once they find a better way to solve the task. This allows the system to eventually outperform its human teacher.

Key Results

IBMARL consistently outperformed demonstration-free MARL (MADDPG) and standard demonstration-based baselines (RLfD, RFT).

  • Speed: On Navigation and Buzzwire, IBMARL's RL actors surpassed the IL policy 2.6x to 4.2x faster than other methods.
  • Success on Hard Tasks: IBMARL was the only method capable of solving the Balance and Transport tasks within the training budget. Baselines often failed to ever see a single reward signal in these highly coordinated environments.
  • Surpassing the Expert: In the Buzzwire and Balance tasks, IBMARL actually achieved higher returns than the original expert demonstrators.
  • Human-in-the-Loop: When trained with real human data (which is noisier than synthetic data), IBMARL still surpassed the human-trained IL policy more than twice as fast as the best baseline.
Results across four VMAS tasks showing IBMARL (red) reaching expert levels much faster than baselines.

Limitations and Future Work

  • Scaling the Arbiter: The current arbiter evaluates $2^N$ combinations. While fine for 2-3 agents, this becomes computationally expensive for large swarms. The authors suggest future work could use greedy searches or random subsets to handle larger teams.
  • Frozen IL Policies: Currently, the IL policy is frozen once RL begins. A "closed-loop" system where a human could provide more demonstrations specifically for states where the RL agent is struggling could be even more efficient.
  • Known Rewards: The method assumes a reward function is already known. In many real-world scenarios, defining the reward is as hard as the task itself; combining IBMARL with reward learning from human preferences is a promising next step.

Summary updated

Abstract

Demonstrations are an effective way to accelerate single-agent reinforcement learning (RL), but extending this idea to multi-agent reinforcement learning (MARL) is non-trivial as it is difficult and often impossible for a single person to teleoperate multiple agents simultaneously. Round-Robin Behavior Cloning (R2BC) addresses this by letting a single demonstrator control one agent at a time while the others execute their current policies. However, the resulting imitation policies are limited by the quality of the demonstrations. We propose Imitation Bootstrapped Multi-Agent Reinforcement Learning (IBMARL), a framework that extends Imitation Bootstrapped Reinforcement Learning (IBRL) to cooperative teams and uses R2BC to train the imitation policies. At each time step, every agent receives one action proposal from its imitation policy and one from its RL policy, and an ensemble of centralized critics selects among the per-agent combinations of the two. A gated imitation penalty, a soft Q-filtered behavior cloning loss, regularizes each RL policy toward its imitation policy only where the critics value the imitation action more, so that after training each agent executes its RL policy alone, without the centralized critics. On four cooperative multi-agent RL benchmarks, IBMARL surpasses its imitation policies $2.6$-$4.2\times$ faster than demonstration-based baselines on two tasks, is the only method to surpass it on the other two, and exceeds the demonstrator itself on two of the four. With real human demonstrations IBMARL still surpasses imitation learning alone, and does so more than twice as fast as the best baseline.
BibTeX
@misc{zhang2026imitation,
  author = {Tony H. Zhang and Connor Mattson and Daniel S. Brown},
  title = {{Imitation Bootstrapped Multi-Agent Reinforcement Learning}},
  year = {2026},
  eprint = {2611.18543},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18543},
}
Figure