Traditional machine learning treats hyperparameter optimization (HPO) as an expensive meta-loop. Practitioners choose a learning rate, weight decay, or dropout probability, run the entire training process to convergence, evaluate the model on a validation set, and repeat. This "black-box" approach assumes that the optimal hyperparameter values are static constants throughout training.
However, as a model navigates the complex, shifting geometry of a loss landscape, its optimal training environment changes. Online hyperparameter optimization shifts the paradigm: instead of searching for a single fixed configuration across multiple training runs, it treats hyperparameters as dynamic variables that adapt on the fly within a single training trajectory.
The Non-Stationarity of Training Dynamics
To understand why online HPO works, we must first recognize that fixed hyperparameters are fundamentally suboptimal. Early in training, a model benefits from high learning rates to escape poor local minima and low regularization to rapidly absorb features. Near convergence, it requires fine-grained steps and tighter regularization to prevent overfitting and stabilize weights.
As shown by Matthew MacKay et al. in Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions, dynamic schedules discovered during training consistently outperform the best possible fixed values found through exhaustive offline sweeps. The optimal configuration for regularization or data augmentation is non-stationary and depends entirely on the model's current state of convergence.
Furthermore, different hyperparameters often exhibit complementary tuning phases. In Accelerating and Improving AlphaZero Using Population Based Training, Ti-Rong Wu et al. observed a "hand-off" behavior where one training variable stabilizes while another is aggressively adjusted. This confirms that the optimal training trajectory requires non-simultaneous shifts across different components of the optimization objective—a feat impossible with static schedules.
Gradient-Based Adaptation and Hypergradients
The most mathematically direct way to adjust hyperparameters mid-training is to compute the gradient of a validation loss with respect to the hyperparameters themselves—a hypergradient. This setup is naturally framed as a bilevel optimization problem: the inner loop updates the model weights on training data, while the outer loop updates the hyperparameters on validation data.
In Forward and Reverse Gradient-Based Hyperparameter Optimization, Luca Franceschi formalized how these hypergradients can be computed dynamically during training rather than waiting for full convergence. To make this practical, online algorithms interleave standard weight updates with gradient-based hyperparameter updates.
[Training Data] ---> Compute Loss ---> Update Model Weights
|
(Interleaved Loop)
v
[Validation Data] -> Compute Loss ---> Compute Hypergradient ---> Update Hyperparameters
For instance, Jelena Luketina et al. introduced the $T_1 - T_2$ method in Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters, demonstrating that updating regularization parameters concurrently with weights avoids the "hysteresis effect" (where current weights are trapped by past suboptimal hyperparameters). To scale this to high-dimensional spaces, Ross M Clarke leveraged the Implicit Function Theorem in Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation, computing exact hypergradients directly from the weight-update rule in a single pass. This allows the model to steer its own configuration trajectory with minimal computational overhead.
Population-Based and Evolutionary Schedules
When hyperparameters are discrete (such as batch size or the number of active participants in a network) or their loss landscapes are non-differentiable, gradient-based methods fail. Evolutionary and population-based approaches solve this by maintaining a pool of parallel learners that explore different hyperparameter spaces simultaneously.
Instead of running these populations independently to completion, online evolutionary methods periodically "hot-swap" parameters. If a learner falls behind in performance, its weights are overwritten by a more successful peer, but it inherits a slightly mutated variation of that peer's hyperparameters.
Su Young Lee et al. utilized this mechanism in Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update to dynamically balance learning speed and stability. Because the synchronization and mutation happen periodically during a single training run, the population collectively traces out an optimal, time-varying hyperparameter schedule. This evolutionary competition allows agents to bypass performance plateaus without requiring dozens of isolated, full-length training executions.
Control Theory and Sequential Decision Making
A third paradigm treats online HPO as a closed-loop control problem or a sequential decision task, adjusting parameters based on real-time performance feedback without relying on explicit hypergradients or large populations.
For example, Kevin Bache et al. formulated hyperparameter selection as a multi-armed bandit problem in Hot Swapping for Online Adaptation of Optimization Hyperparameters, using an upper confidence bound algorithm to evaluate and swap learning rates on a per-minibatch basis. This allows the training process to respond to the local geometry of the loss surface in real time.
Alternatively, classical control loops can be mapped directly onto training dynamics. Siyuan Zhang et al. in A PID Controller Approach for Adaptive Probability-dependent Gradient Decay in Model Calibration used a Proportional-Integral-Derivative (PID) controller to dynamically tune optimization parameters based on validation feedback, correcting for miscalibration trends as they emerge.
From an online optimization standpoint, Ashok Cutkosky et al. developed Mechanic: A Learning Rate Tuner, which frames learning rate selection as an online convex optimization problem designed to minimize regret against the best fixed scalar in hindsight. By treating hyperparameter selection as a secondary optimization task, the system adaptively scales step sizes without manual intervention or proxy experiments.
Key Foundational Work
Dynamically Adapting Kernels in Support Vector Machines (1998) — contributed 1 concept elements to this direction.