MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models

Daniël Willemsen, Mario Coppola, Guido C. H. E. de Croon

TL;DR

This paper introduces MAMBPO, a novel multi-agent reinforcement learning algorithm that uses learned world models to significantly improve sample efficiency. By training a predictive model of the environment, MAMBPO can generate synthetic data, allowing robots to learn complex behaviors with far fewer real-world interactions, a crucial step towards enabling practical real-life multi-robot learning.

What Problem Does This Paper Address?

Training multi-robot systems using reinforcement learning (RL) in the real world is incredibly challenging due to the high cost of time, potential for hardware damage, and the "reality gap" between simulation and reality. While simulations can mitigate some of these issues, they often don't perfectly translate to real-world performance. The core problem this paper tackles is the sample inefficiency of current multi-agent RL algorithms, meaning they require an excessive number of trials to learn effective behaviors. This inefficiency is a major bottleneck preventing the deployment of RL in practical, real-world multi-robot applications.

Methods

The paper proposes MAMBPO (Multi-Agent Model-Based Policy Optimization), an algorithm that combines model-based RL with the Centralized Learning for Decentralized Execution (CLDE) framework.

Here's a breakdown of the key components:

  • CLDE Framework: This framework allows agents to learn using centralized information during training (e.g., shared experiences, global state) but act independently in a decentralized manner during execution. This is crucial for real-world multi-robot systems where communication might be limited or unreliable.
  • Learned World Model: MAMBPO learns a predictive model of the environment's dynamics. This model takes the current joint observation and action of all agents and predicts the next observation and reward. To combat "model bias" (where agents exploit inaccuracies in the learned model), an ensemble of stochastic neural networks is used.
  • Model-Based Policy Optimization (MBPO) Adaptation: The core idea is to use the learned world model to generate synthetic training data. This generated data is then used to train the agents' policies, significantly reducing the need for real-world interactions. This is an adaptation of the single-agent MBPO algorithm.
  • Multi-Agent Soft Actor-Critic (MASAC): For policy optimization, MAMBPO uses MASAC, a CLDE variant of the Soft Actor-Critic algorithm. MASAC incorporates a maximum entropy objective, which encourages exploration and further improves stability and sample efficiency.

In essence, MAMBPO trains a world model on real data, uses that model to generate more data, and then trains the agents' policies on this combined dataset.

Datasets

The paper does not introduce new benchmark datasets. Instead, it uses two benchmark domains from the multi-agent particle environments benchmark suite:

  • Cooperative Navigation: Agents must simultaneously cover three target locations while avoiding collisions.
  • Cooperative Predator-Prey: Three agents (predators) must catch a prey, which uses a heuristic to evade them.

These domains are modified to allow for continuous actions, which is a common requirement for many real-world robotic tasks.

Key Insights and Ideas

  • World Models as Sample Efficiency Boosters: The central idea is that learning a predictive model of the environment allows for the generation of abundant synthetic data. This synthetic data can then be used to train policies much more efficiently than relying solely on real-world interactions.
  • CLDE for Practicality: The CLDE framework is essential for making model-based multi-agent RL practical. It allows for the benefits of centralized learning (like using a global world model) while ensuring agents can operate autonomously in real-world scenarios.
  • Ensembles Mitigate Model Bias: Using an ensemble of neural networks for the world model is a key technique to reduce model bias, a known pitfall in model-based RL that can lead to suboptimal policies.
  • Maximum Entropy for Exploration: Incorporating a maximum entropy objective in the policy optimization (MASAC) helps agents explore more effectively, leading to more robust and sample-efficient learning.
  • Model Quality vs. Agent Performance: The paper investigates the correlation between the learned world model's accuracy (measured by R²) and the agents' performance. They observe that while model quality plateaus, agent performance can continue to improve, suggesting that model quality isn't always the sole limiting factor, but can be in some domains (like predator-prey).

Key Results

MAMBPO demonstrated significant improvements in sample efficiency compared to its model-free counterpart, MASAC.

  • Sample Efficiency: MAMBPO reached performance levels comparable to MASAC in significantly fewer episodes. Specifically, it required 1.7x fewer episodes in cooperative navigation and 3.7x fewer episodes in cooperative predator-prey to achieve similar cumulative rewards.
  • Success Rates: In the cooperative navigation task, MAMBPO achieved a 16.3 percentage point higher success rate halfway through training and a 11.2 percentage point higher success rate after 5000 episodes. In the predator-prey task, MAMBPO reached a 98.8% success rate after 5000 episodes, compared to 76.3% for MASAC.
  • Model Performance: The learned world model showed reasonable accuracy, with R² values for reward prediction being very close to zero bias. However, the model struggled more with predicting velocities and rewards in the predator-prey domain, likely due to its sparse and non-smooth nature.

Limitations and Future Work

  • Thousands of Trials Still Needed: While MAMBPO significantly improves sample efficiency, it still requires thousands of trials to achieve desirable performance. This indicates that further advancements are needed for truly real-life online learning.
  • Improving Model Learning: The authors suggest improving model learning speed and quality through techniques like regularization or attention mechanisms.
  • Extending to Other Algorithms: Exploring model-based enhancements for other multi-agent RL algorithms (e.g., Attention-Actor-Critic, Q-MIX) is a promising direction.
  • Scaling to Complex Tasks: Investigating how these algorithms can be scaled to perform in more complex, real-world tasks is crucial.
  • Sim-to-Real Transfer: Combining sim-to-real learning with model-based RL could allow agents to start learning in the real world without needing to begin from scratch.

Summary updated

Abstract

Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample efficiency. This research thus investigates the use of learned world models to improve sample efficiency. We present a novel multi-agent model-based RL algorithm: Multi-Agent Model-Based Policy Optimization (MAMBPO), utilizing the Centralized Learning for Decentralized Execution (CLDE) framework. CLDE algorithms allow a group of agents to act in a fully decentralized manner after training. This is a desirable property for many systems comprising of multiple robots. MAMBPO uses a learned world model to improve sample efficiency compared to model-free Multi-Agent Soft Actor-Critic (MASAC). We demonstrate this on two simulated multi-robot tasks, where MAMBPO achieves a similar performance to MASAC, but requires far fewer samples to do so. Through this, we take an important step towards making real-life learning for multi-robot systems possible.
BibTeX
@inproceedings{willemsen2021mambpo,
  author = {Daniël Willemsen and Mario Coppola and Guido C. H. E. de Croon},
  title = {{MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9635836},
}
Figure