Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimization

Daphne Theodorakopoulos, Marcel Wever, Marius Lindauer

TL;DR

HPI-ParEGO is a new multi-objective optimization (MOO) framework that dynamically identifies which hyperparameters are most important for specific objective trade-offs during the search. By automatically pruning the search space to focus on these influential parameters, it achieves faster convergence and higher-quality Pareto fronts compared to standard state-of-the-art optimizers.

What Problem Does This Paper Address?

Hyperparameter Optimization (HPO) is usually treated as a single-objective task: maximize accuracy. However, real-world deployment often requires balancing conflicting goals, such as high accuracy versus low latency or minimal energy consumption. This is known as Multi-Objective Optimization (MOO).

The challenge is that most MOO algorithms treat every hyperparameter as equally important throughout the entire search. In reality, a hyperparameter like "learning rate" might be critical for accuracy, while "number of layers" might be the dominant factor for inference time. As an optimizer explores different trade-offs (e.g., moving from a "fast but less accurate" model to a "slow but highly accurate" one), the set of hyperparameters that actually matter changes. Ignoring this "dynamic importance" makes the search process inefficient, especially in high-dimensional spaces.

Methods

The authors introduce HPI-ParEGO, which integrates Hyperparameter Importance (HPI) directly into the optimization loop.

  1. Base Algorithm (ParEGO): It starts with ParEGO, a Bayesian optimization method that converts multiple objectives into a single "scalarized" objective using random weightings. This allows the use of standard surrogate models (like Random Forests).
  2. Dynamic HPI with HyperSHAP: At each step, the algorithm uses HyperSHAP—a method based on Shapley values—to calculate how much each hyperparameter contributes to the predicted performance gain for the current objective weighting.
  3. Configuration Space Subsampling: Based on these importance scores, the algorithm identifies a subset of "important" hyperparameters responsible for a target percentage (e.g., 80%) of the performance gain.
  4. Adaptive Pruning: Unimportant hyperparameters are temporarily fixed to their current best ("incumbent") values. The optimizer then focuses its acquisition function solely on the reduced, high-impact search space.
  5. Symmetric Thresholding: To prevent getting stuck in local optima, the authors use a three-phase schedule:
  6. Initialization: Explore the full space to learn initial HPI.
  7. Reduction: Prune the space to accelerate progress.
  8. Convergence: Re-open the full space for final fine-tuning.

Datasets

The researchers evaluated their method on two main suites:

  • PyMOO: A collection of well-known synthetic benchmark functions (ZDT1, ZDT2, ZDT3, ZDT4, ZDT6) used to test the fundamental mechanics of MOO algorithms.
  • YAHPO-Gym: A surrogate-based benchmark for real-world HPO. They specifically used 13 tasks from LCBench (multi-layer perceptrons) and 13 tasks from rbv2_ranger (random forests), focusing on trade-offs like accuracy vs. memory or accuracy vs. time.

Key Insights and Ideas

  • HPI is a Moving Target: The "importance" of a hyperparameter is not a static property of the dataset; it is a function of the specific trade-off the user currently cares about.
  • Dimensionality Reduction is the Key to Speed: By fixing unimportant parameters, the surrogate model becomes more effective, and the acquisition function (which decides where to sample next) has a much easier job navigating the search space.
  • Incumbents Matter: The ablation study revealed that fixing unimportant parameters to their current best values (incumbents) is significantly more effective than fixing them to default values or random values.
  • The "Symmetric" Strategy: You can't prune forever. Re-opening the search space in the final phase allows the optimizer to perform "fine-tuning" on the parameters it previously ignored, leading to a more refined Pareto front.

Key Results

  • Faster Convergence: On PyMOO tasks, HPI-ParEGO began outperforming the standard ParEGO baseline after only ~30% of the total trial budget.
  • Superior Pareto Fronts: In the LCBench tasks, HPI-ParEGO consistently achieved lower "Hypervolume Regret" (a measure of how close the found solutions are to the true optimal trade-offs) than ParEGO, NSGA-II, and MO-TPE.
  • Robustness: While MO-TPE performed well in the early stages of some tasks, HPI-ParEGO was more robust across different types of benchmarks, eventually overtaking other methods in the final stages of the search.

Limitations and Future Work

  • Surrogate Reliability: The HPI calculation depends on the accuracy of the surrogate model. In the very early stages of optimization (low data), HPI estimates might be noisy.
  • Computational Overhead: Calculating Shapley values adds a small computational cost to each iteration. While this is usually negligible compared to the cost of training an ML model, it could be a factor for extremely cheap-to-evaluate functions.
  • High-Dimensional Scaling: The authors note that for pipelines with hundreds of hyperparameters (like Auto-Sklearn), the overhead of HyperSHAP might become a bottleneck.
  • Future Directions: The authors suggest exploring "meta-learning" to bring in HPI priors from previous tasks or using multi-fidelity optimization (e.g., training on small subsets of data) to speed up the HPI learning phase even further.

Summary updated

Abstract

Choosing a suitable ML model is a complex task that can depend on several objectives, e.g., accuracy, model size, fairness, inference time, or energy consumption. In practice, this requires trading off multiple, often competing, objectives through multi-objective optimization (MOO). However, existing MOO methods typically treat all hyperparameters as equally important, overlooking that hyperparameter importance (HPI) can vary significantly depending on the trade-off between objectives. We propose a novel dynamic optimization approach that prioritizes the most influential hyperparameters based on varying objective trade-offs during the search process, which accelerates empirical convergence and leads to better solutions. Building on prior work on HPI for MOO post-analysis, we now integrate HPI, calculated with HyperSHAP, into the optimization. For this, we leverage the objective weightings naturally produced by the MOO algorithm ParEGO and adapt the configuration space by fixing the unimportant hyperparameters, allowing the search to focus on the important ones. Eventually, we validate our method with diverse tasks from PyMOO and YAHPO-Gym. Empirical results demonstrate improvements in convergence speed and Pareto front quality compared to baselines.
BibTeX
@misc{theodorakopoulos2026dynamic,
  author = {Daphne Theodorakopoulos and Marcel Wever and Marius Lindauer},
  title = {{Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimization}},
  year = {2026},
  eprint = {2601.03166},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.03166},
}
Figure