Mix&Match - Agent Curricula for Reinforcement Learning

Wojciech Marian Czarnecki, Siddhant M. Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Simon Osindero, Nicolas Heess, Razvan Pascanu

Introduction

Training sophisticated Reinforcement Learning (RL) agents—those with large memory requirements (LSTMs) or massive action spaces—is notoriously difficult. Often, these models are so complex that they fail to learn anything at all in the early stages of training, a problem known as the "cold start" issue.

Traditionally, researchers use task-based curricula, where the environment is modified to be easier at first (e.g., a smaller maze). However, this requires fine-grained control over the environment's generation process and deep intuition about what makes a task "simple" for a neural network. Mix & Match (M&M) proposes a more general alternative: keep the task constant but vary the complexity of the agent. M&M creates a curriculum over agent architectures, gradually shifting control from a simple, fast-learning model to a complex, high-performance one.

The Mix & Match Framework

The core of Mix & Match is a sequence of agents $(\pi_1, \dots, \pi_K)$ ordered by increasing complexity.

The Mix & Match framework overview, showing the interaction between the environment and a curriculum of agents.

The framework operates through three primary mechanisms:

  1. Policy Mixing: The framework defines a "composite" policy ($\pi_{mm}$) which is a weighted mixture of the constituent agents. At the start of training, the weight ($\alpha$) favors the simplest agent ($\pi_1$). Over time, $\alpha$ shifts to favor the complex target agent ($\pi_K$).
  2. Knowledge Transfer (Matching): Simply mixing policies isn't enough; the complex agent needs to learn why the simple agent is making certain choices. M&M uses a distillation loss (KL-divergence) to encourage the complex agent to match the probability distribution of the simpler agent's actions.
  3. Automated Curriculum via PBT: Rather than hand-tuning the schedule for $\alpha$, the authors use Population Based Training (PBT). PBT treats $\alpha$ as a hyperparameter that evolves based on the agent's actual performance, allowing the curriculum to speed up or slow down automatically.

Architectural Implementation

M&M allows the agent to explore the environment using a simple, robust policy while the complex policy "watches" and learns, effectively decoupling the difficulty of exploration from the complexity of the final model.

Detailed architecture showing shared CNN/LSTM layers and the mixing of policy and value functions.

A key insight is that architectural constraints can be used as a "crutch" for learning. A reactive, feed-forward agent learns basic movement faster than a recurrent (LSTM) agent. By starting with the former, the latter doesn't have to discover "how to walk" from scratch. This is implemented by sharing lower-level features (like CNN layers) and mixing the outputs of different policy heads.

Experimental Results

The researchers evaluated M&M using DeepMind Lab, a challenging 3D first-person environment, across navigation, combat, and multitask scenarios.

  • Massive Action Spaces: In 3D tasks, the M&M agent with 756 actions reached higher performance and learned significantly faster than agents trained directly on the large action space. The "small" action space (9 actions) provided a dense signal for exploration, which the "big" action space then inherited.
  • Architectural Gains: When transitioning from a Feed-Forward (FF) core to an LSTM core, M&M achieved a 20% increase in human-normalized performance over tasks requiring memory, compared to training an LSTM from scratch.
  • Multitask Stability: In multitask settings, M&M prevented the "winner-take-all" problem. By using specialized "mentor" agents for each task, the central multitask agent maintained a much higher minimum performance across all levels.

Conclusion

Mix & Match provides a powerful framework for bootstrapping complex RL agents. While it requires defining a sensible sequence of agents and carries some additional computational cost due to PBT and multiple models, the gains in convergence speed and final performance—especially in large action spaces—are significant. Future work may extend these principles to value-based methods and more automated agent sequencing.

Summary updated

Abstract

We introduce Mix&Match (M&M) - a training framework designed to facilitate rapid and effective learning in RL agents, especially those that would be too slow or too challenging to train otherwise. The key innovation is a procedure that allows us to automatically form a curriculum over agents. Through such a curriculum we can progressively train more complex agents by, effectively, bootstrapping from solutions found by simpler agents. In contradistinction to typical curriculum learning approaches, we do not gradually modify the tasks or environments presented, but instead use a process to gradually alter how the policy is represented internally. We show the broad applicability of our method by demonstrating significant performance gains in three different experimental setups: (1) We train an agent able to control more than 700 actions in a challenging 3D first-person task; using our method to progress through an action-space curriculum we achieve both faster training and better final performance than one obtains using traditional methods. (2) We further show that M&M can be used successfully to progress through a curriculum of architectural variants defining an agents internal state. (3) Finally, we illustrate how a variant of our method can be used to improve agent performance in a multitask setting.
BibTeX
@inproceedings{czarnecki2018mix,
  author = {Wojciech Marian Czarnecki and Siddhant M. Jayakumar and Max Jaderberg and Leonard Hasenclever and Yee Whye Teh and Simon Osindero and Nicolas Heess and Razvan Pascanu},
  title = {{Mix\&Match - Agent Curricula for Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2018},
  url = {https://openreview.net/forum?id=H1EpW3ZubB},
}
Figure