Abstract
Hyperparameter optimization (HPO) is a critical challenge in Deep Reinforcement Learning (RL) due to the non-stationary nature of the training process. Traditional methods like Population-Based Training (PBT) are computationally expensive, while Bayesian Optimization often fails to adapt to changing training dynamics. We present ULTHO (Ultra-LightweighT HPO), a framework that treats HPO as a hierarchical multi-armed bandit problem. ULTHO optimizes hyperparameters "on the fly" during a single training run, achieving state-of-the-art performance across diverse benchmarks including ALE and Procgen with minimal computational overhead.
Introduction
In Deep RL, the optimal hyperparameters (HPs)—such as learning rate, batch size, or entropy coefficients—often change as the agent's policy evolves. This non-stationarity makes static HPO insufficient. However, existing dynamic HPO solutions typically require massive compute clusters (PBT) or complex meta-gradient calculations. ULTHO addresses this "billion-dollar problem" by providing a plug-and-play, resource-efficient alternative that relies solely on the agent's reward signal.
Methodology: Hierarchical Multi-Armed Bandits
ULTHO frames the HPO task as a Multi-Armed Bandit with Clustered Arms (MABC). To navigate the high-dimensional HP space efficiently, it employs a two-tier hierarchy:
- Tier 1 (Inter-cluster): The algorithm first selects which category of hyperparameter to adjust (e.g., Learning Rate vs. Batch Size).
- Tier 2 (Intra-cluster): It then chooses a specific value from a predefined discrete set within that category.
Selection is guided by the Upper Confidence Bound (UCB) algorithm, which balances exploration of new HP configurations with exploitation of those that have historically yielded higher mean episode returns within a sliding window.
An extension, Relay-ULTHO, identifies the most influential "Cluster of Interest" (COI) to focus optimization efforts, further stabilizing training in environments with lower non-stationarity.
Experimental Evaluation
The authors evaluated ULTHO across several standard RL benchmarks, including the Arcade Learning Environment (ALE), MiniGrid, and PyBullet. A primary focus was the Procgen benchmark, which consists of 16 procedurally generated environments designed to test generalization and handle high non-stationarity.
Results and Discussion
ULTHO demonstrates significant performance and efficiency gains over vanilla PPO and established HPO methods. In the Procgen benchmark, ULTHO outperformed the PPO baseline in 15 out of 16 environments.
Key findings include:
- Efficiency: ULTHO achieves higher aggregate scores than PBT and PB2 while using significantly fewer environment steps (e.g., reaching in 25M steps what baselines often require 100M+ to achieve).
- Critical Hyperparameters: Analysis revealed that the "Number of Update Epochs" (NUE) is a vital lever for performance in dynamic environments, frequently selected by the ULTHO scheduler.
- Robustness: The algorithm is relatively insensitive to its internal UCB exploration coefficients, facilitating a "set-and-forget" implementation.
Conclusion
ULTHO provides a lightweight, effective solution for dynamic HPO in RL. By using a hierarchical bandit approach, it avoids the curse of dimensionality and adapts to the agent's needs throughout training. Future work includes extending the search to continuous hyperparameter spaces and exploring alternative bandit strategies to further enhance stability in noisy environments.
Summary updated