Dynamic Hyperparameter Optimization for Reinforcement Learning

38 concepts27 papers1 proposal

In traditional supervised learning, hyperparameter optimization is typically treated as a static pre-training task. We search for a single optimal configuration—such as a fixed learning rate, batch size, or regularization weight—and keep it constant or follow a predetermined decay schedule throughout training. This approach assumes that the underlying data distribution remains stationary.

In reinforcement learning (RL), this assumption completely breaks down. As Parker-Holder et al. point out, RL is inherently non-stationary because an agent's actions directly determine its future experiences. As the agent learns and its policy evolves, it encounters entirely new data distributions. This feedback loop causes the underlying optimization landscape to deform over time.

Empirical work by Mohan et al. confirms that the optimal hyperparameter landscape shifts drastically over the course of a single training run. Consequently, any static hyperparameter configuration is fundamentally suboptimal; it fails to adapt to the agent's changing needs as it transitions from broad exploration to fine-tuned exploitation (Paul et al.).

The Shifting Algorithmic Landscape

To understand why static configurations fail, we must look at how an agent's ideal parameters change across different phases of training.

Exploration vs. Exploitation

Early in training, an agent needs high exploration noise to discover high-reward states. Later, as the policy begins to converge, this noise must be suppressed to allow for precise policy refinement. As Wang et al. demonstrate, a fixed exploration parameter either cripples early discovery or destabilizes late-stage convergence.

The Trap of "Early Winners"

The ideal step size is also a moving target. Afshar et al. show that standard optimistic multi-armed bandit algorithms struggle with hyperparameter selection because they become biased toward "early winners"—learning rates that excelled during initial exploration but are disastrously large for late-stage fine-tuning.

Model Maturity in Model-Based RL

In model-based reinforcement learning, the agent plans its actions using a learned dynamics model of the environment. Zhang et al. show that the optimal planning horizon must evolve alongside model maturity. Early on, when the dynamics model is highly inaccurate, a short planning horizon prevents compounding errors. As the model becomes reliable, the horizon must expand to enable long-term strategic planning.

Mechanisms for Online Adaptation

If the optimal hyperparameter configuration is a trajectory rather than a single point, how do we track it without incurring prohibitive computational costs? Modern Automated RL (AutoRL) frameworks use several core mechanisms to optimize hyperparameters on the fly.

Meta-Gradients and Self-Tuning

One elegant approach is to make the hyperparameters themselves differentiable, allowing the agent to compute "meta-gradients" to tune its own parameters during training. For instance, Zahavy et al. introduced the Self-Tuning Actor-Critic (STAC) framework. By making off-policy operators differentiable, the authors scaled self-tuning to handle dozens of internal parameters simultaneously, bypassing the need for expensive manual grid searches.

Population-Based Evolution

Instead of optimizing a single agent, population-based methods maintain an ensemble of agents training concurrently. These agents periodically evaluate each other, allowing poorly performing individuals to copy the weights and mutate the hyperparameters of top performers.

Franke et al. developed SEARL, which dramatically improves the sample efficiency of this process by sharing a single replay memory across the entire population. Similarly, Wan et al. combined trust-region Bayesian optimization with generational distillation to smoothly evolve both neural architectures and hyperparameters simultaneously, mitigating the brittleness typical of RL training runs.

Meta-MDPs: Learning to Optimize

Another powerful framing treats hyperparameter selection as a sequential decision-making problem in its own right—a Meta-MDP. Here, the "state" is the current condition of the agent's policy or environment, and the "action" is the choice of hyperparameter for the next training epoch.

Occorso et al. and Sabbioni et al. utilize this framing to dynamically adjust step sizes and trust regions. By training a meta-policy via offline reinforcement learning, the system learns to predict the optimal learning dynamics even for entirely unseen tasks and policy architectures.

On-the-Fly Ensemble Selection

Rather than continuously mutating parameters, some frameworks maintain an ensemble of distinct hyperparameter configurations and dynamically route data or targets to the most effective one. Vincent et al. developed the Adaptive Q-Network (AdaQN), which trains an ensemble of Q-functions with different configurations and dynamically promotes the member with the lowest approximation error to serve as the shared target. This allows the system to seamlessly transition between configurations as the agent gains experience, without requiring additional environment samples.

Summary

The shift from static hyperparameter optimization to dynamic scheduling reflects a deeper realization: reinforcement learning is not a standard optimization problem, but a highly fluid, non-stationary process. By treating hyperparameters as dynamic trajectories managed via meta-gradients, evolutionary populations, or meta-MDPs, modern AutoRL frameworks allow agents to navigate their own evolving learning curves, unlocking unprecedented stability and sample efficiency.

Key Foundational Work

High-Dimensional Continuous Control Using Generalized Advantage Estimation John Schulman (2016) — contributed 1 concept elements to this direction.
Hyperparameter Optimization for Tracking With Continuous Deep Q-Learning (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_hyperparameter_scheduling_00cb98c1,
  title = {dynamic hyperparameter scheduling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-hyperparameter-scheduling/rdr_00cb98c1a8724d20b3a3ea836edc2666},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_hyperparameter_scheduling_00cb98c1_v1,
  title = {dynamic hyperparameter scheduling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-hyperparameter-scheduling/rdr_00cb98c1a8724d20b3a3ea836edc2666/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake