TL;DR
OMPAC (Online Meta-learning by Parallel Algorithm Competition) is a simple yet powerful method for automatically tuning reinforcement learning hyperparameters during training. By running multiple agents in parallel and using a "survival of the fittest" approach to select and mutate meta-parameters, it achieves significant performance gains—up to 84% in Tetris and 62% in Atari games—while eliminating the need for exhaustive manual grid searches.
What Problem Does This Paper Address?
Reinforcement learning (RL) is notoriously sensitive to "meta-parameters" like the learning rate ($\alpha$), the discount factor ($\gamma$), and the exploration rate ($\epsilon$ or temperature $\tau$). A value that works well at the start of training might lead to instability or stagnation later on.
Traditionally, researchers find these values through exhaustive grid searches, which are computationally prohibitive for modern deep RL tasks that take days or weeks to train. Furthermore, static meta-parameters are rarely optimal across an entire learning trajectory. The authors address the challenge of online adaptation: how can an agent discover the best settings for its current state of learning without starting from scratch?
Methods
The authors introduce OMPAC, a method inspired by Lamarckian evolution. Unlike Darwinian evolution, where offspring start from a blank slate, OMPAC agents pass on their learned knowledge (neural network weights) to the next generation.
- Parallel Instances: Run $N$ instances (e.g., 12) of an RL algorithm (like Sarsa($\lambda$) or TD($\lambda$)) in parallel.
- Initial Variation: Each instance starts with slightly different meta-parameter values, generated by adding Gaussian noise to a common starting point.
- Competition and Selection: After a fixed number of episodes (a "generation"), instances are evaluated based on their raw task score. The best-performing instance is kept as an "elite."
- Stochastic Universal Sampling (SUS): The remaining slots for the next generation are filled by selecting from the current instances with a probability proportional to their performance.
- Mutation: Gaussian noise is added to the meta-parameters of the selected instances (except the elite) to encourage further exploration of the parameter space.
- Continuity: Crucially, the agents continue learning from their current weights; they do not reset.
The study also utilizes Sigmoid-weighted Linear (SiL) and derivative SiL (dSiL) units as activation functions, which have shown promise in previous RL research.
Datasets
The method was validated on two distinct domains:
- Tetris Variants:
- SZ-Tetris: A challenging version using only S- and Z-shaped blocks, making line clears difficult.
- 10x10 Tetris: Standard Tetris played on a smaller board.
- Atari 2600 (Arcade Learning Environment): The authors focused on three games—Alien, Amidar, and Assault—where previous deep RL baselines had room for improvement.
Key Insights and Ideas
- Dynamic Parameter Trajectories: The experiments revealed that meta-parameters don't just settle; they evolve. For instance, the learning rate ($\alpha$) consistently decreased by an order of magnitude as agents mastered tasks, while the inverse temperature ($\beta$) increased, shifting the agent from exploration to exploitation.
- Objective Duality: OMPAC distinguishes between the learning objective (maximizing discounted rewards) and the task goal (maximizing raw score). This allows the selection process to be guided by what humans actually care about (the score) while the agent optimizes its internal value functions.
- Robustness to "Bad" Starts: One of the most striking findings is OMPAC's ability to recover from poor initial hyperparameter settings. Even when started with parameters that caused standard agents to fail completely, OMPAC successfully navigated the parameter space to find high-performing regions.
Key Results
- Tetris: OMPAC improved the state-of-the-art mean scores by 31% in SZ-Tetris and a massive 84% in 10x10 Tetris.
- Atari: In the three Atari games tested, OMPAC-adapted deep Sarsa($\lambda$) agents improved their mean scores by 62% to 82% compared to fixed-parameter baselines.
- Competitive Performance: The OMPAC scores were generally higher than reported results for DQN and Gorila, and competitive with Double DQN, despite using a simpler base algorithm.
Limitations and Future Work
- Computational Overhead: While OMPAC is efficient in terms of "wall-clock" time if you have a parallel cluster, it still requires $N$ times the total compute cycles of a single agent.
- Selection Frequency: The number of episodes per generation (100 in this study) is a new meta-meta-parameter. While the authors found a value that worked across different tasks, a more principled way to determine this frequency could be beneficial.
- Algorithmic Scope: The study focused on value-based methods. Future work could explore applying OMPAC to policy gradient methods or actor-critic architectures.
Summary updated