TL;DR
This paper introduces Multi-Expert Distillation (MED), a novel approach for training agents that can quickly learn to coordinate with new teammates. Instead of learning from scratch, MED distills knowledge from a diverse set of pre-trained "expert" agents, significantly improving training efficiency and stability.
What Problem Does This Paper Address?
Training agents to effectively collaborate with unknown teammates, known as "ad hoc teamwork," is a significant challenge in multi-agent reinforcement learning (MARL). Traditional methods often require extensive training from scratch with each new teammate, leading to inefficient and unstable learning. This paper focuses on "few-shot coordination," where an agent needs to adapt its behavior rapidly to cooperate with unseen partners over a few episodes. Existing methods like Population-Based Training (PBT) can be inefficient because they train the generalist agent from scratch, failing to leverage the coordination knowledge already present within the population of diverse teammate strategies.
Methods
The core innovation of MED is to treat existing diverse teammate policies within a population as "experts" and distill their knowledge into a single generalist agent. Instead of relying solely on reinforcement learning from scratch, MED employs a supervised learning approach.
Here's how it works:
- Population Generation: A diverse set of high-quality teammate policies is generated using a method called LIPO.
- Across-Episodic Trajectory Sampling: The generalist agent interacts with these expert teammates in various environments. During these interactions, the expert teammates generate "advised actions."
- Supervised Distillation: The generalist agent's transformer-based policy model is trained using these advised actions as targets. The historical context of the interaction (past observations and actions) is used as input to predict the expert's next action. This allows the generalist to learn coordination strategies by mimicking the experts.
The use of a transformer model is crucial, as it enables the agent to analyze teammate cooperation styles and align its behavior with expert strategies by processing sequential game episodes.
Datasets
The paper evaluates MED on three environments that emphasize coordination and policy diversity:
- One-Step Cooperative Matrix Game: A simple game designed to highlight distinct coordination challenges.
- Gridworld MoveBox: A more complex environment where two agents must collaborate to move a box to an exit.
- Overcooked: A recipe-preparation game requiring agents to coordinate actions to serve dishes quickly.
These environments are not new datasets but rather benchmarks used to test the coordination capabilities of MARL agents.
Key Insights and Ideas
- Distillation is More Efficient than Scratch Learning: The paper strongly suggests that learning by mimicking existing expert policies is significantly more efficient than training a generalist agent from scratch. This is a key takeaway for designing more practical ad hoc teamwork agents.
- Leveraging Historical Context is Crucial for Adaptation: The transformer architecture allows MED to effectively use past interaction histories to understand teammate behavior and adapt its own strategy, leading to improved coordination over time.
- Stability Matters: Unlike baseline methods that show high variance in training, MED demonstrates a more stable learning process, making it more reliable for real-world applications.
Key Results
Experiments show that MED achieves:
- Remarkable Learning Speed: MED requires significantly fewer total timesteps to reach comparable or better performance than baseline methods like DDQN, RL², and PEARL, which train from scratch.
- Improved Stability: The training curves for MED are notably more stable compared to the baselines, which exhibit significant variation.
- Effective Adaptation: In the Overcooked environment, MED's performance shows a significant surge in the second episode after initial interactions, demonstrating its ability to adapt its behavior based on learned historical contexts and enhance cooperation.
Limitations and Future Work
- Limitations: While MED shows promise, the paper doesn't explicitly detail limitations beyond the inherent challenges of few-shot coordination. The reliance on a pre-generated population of experts might be a bottleneck if generating diverse and high-quality experts is difficult.
- Future Work: The authors suggest focusing on integrating the benefits of both supervised learning (as used in MED) and reinforcement learning within ad hoc teamwork settings. This could potentially lead to agents that can both learn from experts and further refine their strategies through self-exploration.
Summary updated