Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning

Hui Bai, Ran ChengIEEE Transactions on Emerging Topics in Computational Intelligence 2024 · DOI · OpenReview

TL;DR

This paper introduces Generalized Population-Based Training (GPBT) and Pairwise Learning (PL) to improve hyperparameter optimization (HPO) for Reinforcement Learning (RL). GPBT offers a more flexible framework than traditional PBT, while PL uses a novel pseudo-gradient approach to guide underperforming agents, leading to more robust and efficient HPO in complex RL tasks.

What Problem Does This Paper Address?

Reinforcement Learning (RL) agents learn by interacting with dynamic environments, which necessitates continuous adaptation of their learning parameters (hyperparameters). Hyperparameter Optimization (HPO) is crucial for achieving good performance, but traditional methods can be slow and inefficient, especially in RL where the learning process itself is non-stationary. Population-Based Training (PBT) was a step forward, allowing agents to learn and adapt hyperparameters in parallel. However, PBT tends to favor already high-performing agents, potentially missing out on agents that could become excellent with more exploration. This can lead to premature convergence and suboptimal solutions. The paper aims to address PBT's limitations by introducing a more adaptable and explorative HPO framework for RL.

Methods

The core contribution is the Generalized Population-Based Training (GPBT) framework. GPBT builds on PBT's asynchronous parallel training but offers more flexibility in how agents adapt their hyperparameters. Instead of a fixed strategy, GPBT allows users to define custom adaptation methods.

To further enhance GPBT, the authors introduce Pairwise Learning (PL). PL is a novel HPO method inspired by Stochastic Gradient Descent with Momentum (SGDM). Instead of relying solely on elite agents, PL uses a pairwise comparison between agents. A "slow learner" (underperforming agent) uses a "pseudo-gradient" derived from the performance difference with a "fast learner" (superior agent) to update its hyperparameters. This approach aims to provide more targeted guidance to lagging agents, promoting exploration of potentially overlooked regions in the hyperparameter space. The integration of GPBT and PL results in the GPBT-PL framework.

Datasets

The experiments were conducted using various tasks from the OpenAI Gym environment, a standard benchmark suite for RL. Specifically, they used:

  • On-policy RL tasks: BipedalWalker, Ant, HalfCheetah, InvertedDoublePendulum, Swimmer, and Walker2D.
  • Off-policy RL tasks: Breakout and SpaceInvaders (from the Arcade Learning Environment).

No new benchmarks were introduced; the focus was on evaluating the proposed methods on existing, well-established RL tasks.

Key Insights and Ideas

  • PBT's "winner-take-all" bias can hinder exploration: The tendency of PBT to favor top performers can lead to discarding promising agents too early, limiting the search for truly optimal hyperparameters.
  • Pairwise learning offers targeted guidance: By comparing agents in pairs and using the performance difference to guide underperforming agents, PL provides a more nuanced way to improve hyperparameters than simply copying elite agents.
  • GPBT-PL balances exploration and exploitation effectively: The combination of GPBT's flexibility and PL's guided exploration allows the framework to discover better hyperparameter configurations, especially in complex RL scenarios.
  • Adaptability is key for RL HPO: The non-stationary nature of RL demands HPO methods that can adapt dynamically, which GPBT-PL achieves through its flexible framework and pairwise learning mechanism.
  • Robustness to perturbation intervals is crucial: The performance of PBT-like methods can be sensitive to how often hyperparameters are updated. GPBT-PL shows improved robustness to these changes.

Key Results

  • Superior performance over PBT and PB2: Across a range of OpenAI Gym tasks, GPBT-PL consistently outperformed traditional PBT and its Bayesian-optimized variant (PB2).
  • Significant improvements in complex tasks: For smaller populations (n=4), GPBT-PL showed substantial gains, e.g., +24% and +32% over PBT for Swimmer and Walker2D respectively.
  • Efficiency gains: GPBT-PL achieved high-level performance with fewer workers (5 workers per agent) compared to some baseline methods. For instance, it matched the performance of a hand-tuned IMPALA with 32 workers in SpaceInvaders.
  • Robustness: GPBT-PL demonstrated better resilience to variations in perturbation intervals and hyperparameter ranges compared to PBT and PB2.
  • Scalability: While increasing population size generally helped, GPBT-PL showed a better ability to surpass local optima with larger populations.

Limitations and Future Work

  • Performance on simpler tasks: The paper notes that GPBT-PL's superiority is more pronounced in complex tasks and less apparent on simpler ones.
  • Computational cost with many hyperparameters: While GPBT-PL is generally efficient, optimizing a very large number of hyperparameters simultaneously can still be computationally demanding and may require careful selection of which hyperparameters to tune together.
  • Future work suggestions: The authors suggest exploring EC-based HPO methods that can better navigate high-dimensional and combinatorial search spaces, especially when dealing with a large number of hyperparameters and broad ranges. They also highlight the need for strategies that can discern accurate update directions for hyperparameters in such complex scenarios.

Summary updated

Abstract

Hyperparameter optimization plays a key role in the machine learning domain. Its significance is especially pronounced in reinforcement learning (RL), where agents continuously interact with and adapt to their environments, requiring dynamic adjustments in their learning trajectories. To cater to this dynamicity, the Population-Based Training (PBT) was introduced, leveraging the collective intelligence of a population of agents learning simultaneously. However, PBT tends to favor high-performing agents, potentially neglecting the explorative potential of agents on the brink of significant advancements. To mitigate the limitations of PBT, we present the Generalized Population-Based Training (GPBT), a refined framework designed for enhanced granularity and flexibility in hyperparameter adaptation. Complementing GPBT, we further introduce Pairwise Learning (PL). Instead of merely focusing on elite agents, PL employs a comprehensive pairwise strategy to identify performance differentials and provide holistic guidance to underperforming agents. By integrating the capabilities of GPBT and PL, our approach significantly improves upon traditional PBT in terms of adaptability and computational efficiency. Rigorous empirical evaluations across a range of RL benchmarks confirm that our approach consistently outperforms not only the conventional PBT but also its Bayesian-optimized variant.
BibTeX
@article{bai2024generalized,
  author = {Hui Bai and Ran Cheng},
  title = {{Generalized Population-Based Training for Hyperparameter Optimization in Reinforcement Learning}},
  journal = {IEEE Transactions on Emerging Topics in Computational Intelligence},
  year = {2024},
  doi = {10.1109/tetci.2024.3389777},
}
Figure