Introduction
Training sophisticated Reinforcement Learning (RL) agents—those with large memory requirements (LSTMs) or massive action spaces—is notoriously difficult. Often, these models are so complex that they fail to learn anything at all in the early stages of training, a problem known as the "cold start" issue.
Traditionally, researchers use task-based curricula, where the environment is modified to be easier at first (e.g., a smaller maze). However, this requires fine-grained control over the environment's generation process and deep intuition about what makes a task "simple" for a neural network. Mix & Match (M&M) proposes a more general alternative: keep the task constant but vary the complexity of the agent. M&M creates a curriculum over agent architectures, gradually shifting control from a simple, fast-learning model to a complex, high-performance one.
The Mix & Match Framework
The core of Mix & Match is a sequence of agents $(\pi_1, \dots, \pi_K)$ ordered by increasing complexity.
The framework operates through three primary mechanisms:
- Policy Mixing: The framework defines a "composite" policy ($\pi_{mm}$) which is a weighted mixture of the constituent agents. At the start of training, the weight ($\alpha$) favors the simplest agent ($\pi_1$). Over time, $\alpha$ shifts to favor the complex target agent ($\pi_K$).
- Knowledge Transfer (Matching): Simply mixing policies isn't enough; the complex agent needs to learn why the simple agent is making certain choices. M&M uses a distillation loss (KL-divergence) to encourage the complex agent to match the probability distribution of the simpler agent's actions.
- Automated Curriculum via PBT: Rather than hand-tuning the schedule for $\alpha$, the authors use Population Based Training (PBT). PBT treats $\alpha$ as a hyperparameter that evolves based on the agent's actual performance, allowing the curriculum to speed up or slow down automatically.
Architectural Implementation
M&M allows the agent to explore the environment using a simple, robust policy while the complex policy "watches" and learns, effectively decoupling the difficulty of exploration from the complexity of the final model.
A key insight is that architectural constraints can be used as a "crutch" for learning. A reactive, feed-forward agent learns basic movement faster than a recurrent (LSTM) agent. By starting with the former, the latter doesn't have to discover "how to walk" from scratch. This is implemented by sharing lower-level features (like CNN layers) and mixing the outputs of different policy heads.
Experimental Results
The researchers evaluated M&M using DeepMind Lab, a challenging 3D first-person environment, across navigation, combat, and multitask scenarios.
- Massive Action Spaces: In 3D tasks, the M&M agent with 756 actions reached higher performance and learned significantly faster than agents trained directly on the large action space. The "small" action space (9 actions) provided a dense signal for exploration, which the "big" action space then inherited.
- Architectural Gains: When transitioning from a Feed-Forward (FF) core to an LSTM core, M&M achieved a 20% increase in human-normalized performance over tasks requiring memory, compared to training an LSTM from scratch.
- Multitask Stability: In multitask settings, M&M prevented the "winner-take-all" problem. By using specialized "mentor" agents for each task, the central multitask agent maintained a much higher minimum performance across all levels.
Conclusion
Mix & Match provides a powerful framework for bootstrapping complex RL agents. While it requires defining a sensible sequence of agents and carries some additional computational cost due to PBT and multiple models, the gains in convergence speed and final performance—especially in large action spaces—are significant. Future work may extend these principles to value-based methods and more automated agent sequencing.
Summary updated