Reinforcement learning for online hyperparameter tuning in convex quadratic programming

Jeremy Bertoncini, Alberto De Marchi, Matthias Gerdts, Simon Gottschalk

TL;DR

Researchers have developed a reinforcement learning (RL) framework that dynamically tunes the hyperparameters of a stabilized interior-point solver for convex quadratic programming. By learning an optimal "policy" for adjusting penalty parameters on the fly, the approach significantly speeds up convergence and improves solver robustness, generalizing across problem sizes and types with minimal training overhead.

What Problem Does This Paper Address?

Quadratic Programming (QP) is a fundamental tool in everything from robotics and control to finance. While we have reliable algorithms to solve these problems—like interior-point (IP) methods—their performance is notoriously sensitive to "hyperparameters." These are internal settings (like regularization weights) that dictate how the solver balances feasibility versus optimality.

Traditionally, tuning these parameters is a bit of a "black art," often relying on manual trial-and-error or static heuristics. If the parameters are poorly chosen, the solver might take thousands of unnecessary iterations or fail to converge to a high-accuracy solution. This paper tackles the challenge of online hyperparameter tuning: can we use data-driven methods to adjust these settings while the solver is running to reach the finish line faster?

Methods

The authors combine a stabilized primal-dual interior-point solver with a Reinforcement Learning (RL) agent.

  1. The Solver: They use a two-loop structure. The "outer loop" handles regularization and proximal updates, while the "inner loop" solves a subproblem using Newton’s method.
  2. The RL Agent: Using the Proximal Policy Optimization (PPO) algorithm, the agent observes the current state of the solver and decides how to adjust the regularization parameters ($\delta$) for the next step.
  3. State Representation: To make the agent "smart" regardless of the problem's size or scale, they feed it log-scaled, normalized residuals (measures of how far the current guess is from the solution).
  4. Reward Function: The agent is rewarded for three things:
  5. Reducing the gap between primal and dual residuals (keeping the search "balanced").
  6. Making large jumps toward the solution between iterations.
  7. Finishing the job quickly (an "iteration budget" bonus).
  8. Curriculum Learning: They used a multistage training process, starting with simple goals (balancing residuals) and gradually introducing the performance-based rewards.

Datasets

  • Synthetic Training Set: Randomly generated convex QPs with controlled properties like sparsity, ill-conditioning (difficult numerical properties), and linear dependencies in constraints.
  • Scale 10× Set: A validation set with problems 10 times larger than the training data to test if the RL policy scales.
  • Maros–Mészáros Benchmark: A famous, "real-world" collection of challenging QPs used to test the agent's zero-shot generalization (how well it performs on problems it has never seen before).

Key Insights and Ideas

  • Scale Invariance is Key: By using log-scaled residuals and barrier parameters in the RL state, the authors created a policy that doesn't care if the problem has 20 variables or 200. This allows a "lightweight" model trained on small problems to work on much larger ones.
  • RL as a "Safety Net": The experiments showed that even if the solver's initial manual settings were poorly chosen, the RL agent could "rescue" the process by adapting the parameters dynamically, maintaining performance where static heuristics would fail.
  • Efficiency over Brute Force: Unlike previous attempts (like RLQP) that required days of training on high-end clusters, this framework was trained in just 13 hours on a standard laptop, making it much more accessible for practical research.

Key Results

  • Speedup: On the "Scale 10×" set, the RL-enhanced solver was significantly faster than standard geometric decrease strategies. For 80% of the problems, it achieved a substantial performance gain, sometimes reaching a 16x speedup ($2^4$ ratio) in terms of iterations.
  • Robustness: On the Maros–Mészáros benchmark, the RL policy improved solver robustness by 2% to 7% compared to the best-fixed parameter configurations.
  • Generalization: The agent successfully identified meaningful parameter adjustments for the benchmark set despite those problems having entirely different structures than the random synthetic data used for training.

Limitations and Future Work

  • Inference Overhead: For very small, simple problems that solve in milliseconds, the time it takes for the neural network to "think" (inference time) can actually outweigh the savings in iteration count.
  • Parameter Scope: The current study focused on tuning the regularization weights ($\delta$). The authors suggest that future work could also include the barrier parameter ($\nu$) and inner tolerances ($\epsilon$) in the RL action space.
  • Specific Structures: While the agent generalizes well, the authors believe even better performance could be achieved by training on specific classes of problems, such as those found in Model Predictive Control (MPC) or power systems.

Summary updated

Abstract

Quadratic programming is a workhorse of modern nonlinear optimization, control, and data science. Although regularized methods offer convergence guarantees under minimal assumptions on the problem data, they can exhibit the slow tail-convergence typical of first-order schemes, thus requiring many iterations to achieve high-accuracy solutions. Moreover, hyperparameter tuning significantly impacts on the solver performance but how to find an appropriate parameter configuration remains an elusive research question. To address these issues, we explore how data-driven approaches can accelerate the solution process. Aiming at high-accuracy solutions, we focus on a stabilized interior-point solver and carefully handle its two-loop flow and control parameters. We will show that reinforcement learning can make a significant contribution to facilitating the solver tuning and to speeding up the optimization process. Numerical experiments demonstrate that, after a lightweight training, the learned policy generalizes well to different problem classes with varying dimensions and to various solver configurations.
BibTeX
@misc{bertoncini2025reinforcement,
  author = {Jeremy Bertoncini and Alberto De Marchi and Matthias Gerdts and Simon Gottschalk},
  title = {{Reinforcement learning for online hyperparameter tuning in convex quadratic programming}},
  year = {2025},
  eprint = {2509.07404},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.07404},
}
Figure