Dynamic Algorithm Configuration with Reinforcement Learning

37 concepts28 papers1 proposal

When we train a machine learning model or solve a complex engineering problem, we rely on optimization algorithms to find the best solutions. However, these optimizers themselves depend on hyperparameters—such as step sizes, mutation rates, penalty parameters, and damping factors—that dictate how they explore the search space.

Choosing these parameters is often described as a "dark art" (Iacca et al.). Traditionally, practitioners rely on static values found via expensive grid searches, or hand-designed heuristic schedules (like cosine annealing). But static configurations introduce a fundamental bottleneck: the ideal behavior of an optimizer changes dynamically as the search progresses.

Dynamic Algorithm Configuration (DAC) solves this by treating the execution of an optimizer as a sequential decision-making problem. By pairing the optimizer with a Reinforcement Learning (RL) agent, the algorithm can observe its own progress and adjust its hyperparameters on the fly, adapting to the local topology of the problem space as it is discovered (Ma et al.; Adriaensen et al.).


Framing Optimization as a Markov Decision Process

To understand why this approach works, we must look at how an optimization algorithm can be modeled as an RL environment. In a standard RL setup, an agent takes actions in an environment to maximize a cumulative reward. In DAC, the "environment" is the optimizer itself, and the "state" is a snapshot of the current optimization trajectory.

┌────────────────────────────────────────────────────────┐
│                  Reinforcement Learning Agent          │
└───────────────────────────┬────────────────────────────┘
                            │ Action: Adjust Hyperparameters
                            │ (e.g., step size, mutation rate)
                            ▼
┌────────────────────────────────────────────────────────┐
│               Target Optimization Algorithm            │
│  (e.g., Evolutionary Solver, Gradient Descent, ADMM)   │
└───────────────────────────┬────────────────────────────┘
                            │ State Feedback & Reward
                            │ (e.g., loss reduction, residuals)
                            ▼

For any iterative solver, we can map its components to a Markov Decision Process (MDP):

  • State ($S_t$): Instead of physical coordinates, the state space consists of internal algorithm metrics. This might include the current loss, the rate of objective reduction, population diversity statistics, or primal and dual residuals (Adriaensen et al.).
  • Action ($A_t$): The actions are the modifications made to the hyperparameters for the next iteration or generation cycle, such as scaling factors or crossover rates (Reijnen et al.).
  • Reward ($R_t$): The reward is typically tied to efficiency and solution quality—for example, the immediate reduction in the objective function or a penalty for each computational step taken under a strict time budget (Ruvolo et al.).

By framing the problem this way, the RL agent learns a policy that responds directly to real-time feedback from the optimization landscape.


Why It Works: Intuition and Mechanisms

The core reason DAC outperforms static heuristics comes down to how optimization landscapes vary. A fixed schedule assumes a uniform, predictable path toward a minimum. In reality, an optimizer might encounter a flat plateau, a narrow ridge, or a chaotic valley.

In evolutionary and swarm-based optimization, algorithms must balance exploring new regions with exploiting known good solutions. Giovanni Iacca and Michele Tessari show that an RL agent can discover non-intuitive schedules for mutation rates and step sizes, allowing the solver to dynamically shift its behavior based on the current generation's state (Iacca et al.). For instance, if a swarm algorithm begins to cluster too tightly without improving the objective, the agent can sense this stagnation and temporarily increase exploration parameters—such as the contraction-expansion coefficient—to break out of local minima (Xiong et al.).

Balancing Mathematical Trade-offs

In continuous and non-linear optimization, hyperparameters often control the mathematical balance between different search strategies. Consider Bundle Adjustment, the non-linear least squares backbone of 3D reconstruction. Amir Belder and Ayellet Tal trained a Soft Actor-Critic agent to dynamically adjust the damping factor ($\lambda$) on the fly (Belder et al.). Instead of relying on rigid, hand-crafted Levenberg-Marquardt heuristics, the learned policy continuously balances gradient descent and second-order methods based on the error landscape, accelerating convergence by 3x to 5x.

A similar mechanism operates in distributed optimization frameworks like the Alternating Direction Method of Multipliers (ADMM). Sihan Zeng demonstrates that a Deep Q-Learning policy can adaptively select penalty parameters by tracking the history of primal and dual residuals (Zeng). When residuals diverge, the agent tunes the penalty to force coordination, reducing the total iteration count by up to 59%.


Generalization and Advanced State Representations

A common concern with meta-optimization is whether a learned controller can generalize beyond its training instances. If an RL agent only learns to optimize a single function, the overhead of training the agent outweighs its benefits.

Empirical evidence shows that these learned policies generalize remarkably well to unseen problem domains and larger scales. For example, Paul Ruvolo et al. demonstrated that policies trained under tight computational budgets generalize across completely different problem domains (Ruvolo et al.). Similarly, Robbert Reijnen et al. showed that an RL controller trained on small multi-objective problems can successfully guide evolutionary algorithms through much larger, unseen combinatorial problem sizes (Reijnen et al.).

To achieve this level of generalization, researchers have turned to richer state representations. In multi-objective optimization, the goal is to find a diverse Pareto front of trade-offs. To capture this, Robbert Reijnen et al. introduced a framework that models the evolving population of solutions as a graph in the objective space (Reijnen et al.). By passing this graph through a Graph Neural Network (GNN), the RL agent perceives structural shifts in the optimization frontier, allowing it to make highly precise parameter adjustments regardless of the underlying problem size.

┌───────────────────────────┐
│ Evolving Population       │
│ (Solutions in Obj. Space) │
└─────────────┬─────────────┘
              │
              ▼
┌───────────────────────────┐
│ Graph Representation      │ ──► Processed by GNN
└───────────────────────────┘
              │
              ▼
┌───────────────────────────┐
│ RL Agent perceives        │ ──► Outputs optimal continuous
│ structural topology       │     parameter configurations
└───────────────────────────┘

Broadening the Scope: Meta-Optimization Everywhere

While DAC originated as a tool for classical numerical and evolutionary optimization, the paradigm of using a learned agent to dynamically guide an underlying process has spread across machine learning and engineering:

  • Machine Learning Pipelines: The authors of Efficient Multi-view Clustering via Reinforcement Contrastive Learning utilize a Q-learning agent to monitor cluster compactness and separation, dynamically adjusting training hyperparameters like temperature and confidence thresholds on the fly. In federated learning, Qian Chen et al. use a deterministic policy gradient agent to tune local training epochs and server aggregation weights in real-time to balance personalization and generalization (Chen et al.).
  • Optimizing the Optimizers: Instead of tuning hyperparameters, Carlo Alfano et al. use evolutionary strategies to optimize the functional form of mirror maps within Policy Mirror Descent, replacing hand-designed regularizers with geometries tailored to specific task dynamics (Alfano et al.). Conversely, Jake Grigsby et al. use evolutionary populations to dynamically tune the internal hyperparameters (like replay ratios and discount factors) of actor-critic reinforcement learning agents during training (Grigsby et al.).
  • Interactive Guidance Loops: Beyond pure mathematics, this interactive loop can optimize physical data collection. Shaheer U. Saeed et al. apply RL to dynamically predict optimal acquisition parameters during ultrasound scanning, transforming image registration into an interactive guidance loop that continuously adapts to real-time environmental feedback (Saeed et al.).

Ultimately, whether the target algorithm is an evolutionary scheduler, a distributed power-grid solver, or a deep neural network, the core insight remains the same: optimization is not a static journey. By treating algorithm configuration as a sequential decision-making problem, we can replace rigid human heuristics with flexible, learned policies that adapt to the landscape in real-time.

Key Foundational Work

Étude de différentes combinaisons de comportements adaptatives (2006) — contributed 1 concept elements to this direction.
Optimization on a Budget: A Reinforcement Learning Approach (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_hyperparameter_adaptation_b3b1b11a,
  title = {dynamic hyperparameter adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-hyperparameter-adaptation/rdr_b3b1b11a25ad4b5191e84798072d3f81},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_hyperparameter_adaptation_b3b1b11a_v1,
  title = {dynamic hyperparameter adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-hyperparameter-adaptation/rdr_b3b1b11a25ad4b5191e84798072d3f81/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake