TL;DR
HPI-ParEGO is a new multi-objective optimization (MOO) framework that dynamically identifies which hyperparameters are most important for specific objective trade-offs during the search. By automatically pruning the search space to focus on these influential parameters, it achieves faster convergence and higher-quality Pareto fronts compared to standard state-of-the-art optimizers.
What Problem Does This Paper Address?
Hyperparameter Optimization (HPO) is usually treated as a single-objective task: maximize accuracy. However, real-world deployment often requires balancing conflicting goals, such as high accuracy versus low latency or minimal energy consumption. This is known as Multi-Objective Optimization (MOO).
The challenge is that most MOO algorithms treat every hyperparameter as equally important throughout the entire search. In reality, a hyperparameter like "learning rate" might be critical for accuracy, while "number of layers" might be the dominant factor for inference time. As an optimizer explores different trade-offs (e.g., moving from a "fast but less accurate" model to a "slow but highly accurate" one), the set of hyperparameters that actually matter changes. Ignoring this "dynamic importance" makes the search process inefficient, especially in high-dimensional spaces.
Methods
The authors introduce HPI-ParEGO, which integrates Hyperparameter Importance (HPI) directly into the optimization loop.
- Base Algorithm (ParEGO): It starts with ParEGO, a Bayesian optimization method that converts multiple objectives into a single "scalarized" objective using random weightings. This allows the use of standard surrogate models (like Random Forests).
- Dynamic HPI with HyperSHAP: At each step, the algorithm uses HyperSHAP—a method based on Shapley values—to calculate how much each hyperparameter contributes to the predicted performance gain for the current objective weighting.
- Configuration Space Subsampling: Based on these importance scores, the algorithm identifies a subset of "important" hyperparameters responsible for a target percentage (e.g., 80%) of the performance gain.
- Adaptive Pruning: Unimportant hyperparameters are temporarily fixed to their current best ("incumbent") values. The optimizer then focuses its acquisition function solely on the reduced, high-impact search space.
- Symmetric Thresholding: To prevent getting stuck in local optima, the authors use a three-phase schedule:
- Initialization: Explore the full space to learn initial HPI.
- Reduction: Prune the space to accelerate progress.
- Convergence: Re-open the full space for final fine-tuning.
Datasets
The researchers evaluated their method on two main suites:
- PyMOO: A collection of well-known synthetic benchmark functions (ZDT1, ZDT2, ZDT3, ZDT4, ZDT6) used to test the fundamental mechanics of MOO algorithms.
- YAHPO-Gym: A surrogate-based benchmark for real-world HPO. They specifically used 13 tasks from
LCBench(multi-layer perceptrons) and 13 tasks fromrbv2_ranger(random forests), focusing on trade-offs like accuracy vs. memory or accuracy vs. time.
Key Insights and Ideas
- HPI is a Moving Target: The "importance" of a hyperparameter is not a static property of the dataset; it is a function of the specific trade-off the user currently cares about.
- Dimensionality Reduction is the Key to Speed: By fixing unimportant parameters, the surrogate model becomes more effective, and the acquisition function (which decides where to sample next) has a much easier job navigating the search space.
- Incumbents Matter: The ablation study revealed that fixing unimportant parameters to their current best values (incumbents) is significantly more effective than fixing them to default values or random values.
- The "Symmetric" Strategy: You can't prune forever. Re-opening the search space in the final phase allows the optimizer to perform "fine-tuning" on the parameters it previously ignored, leading to a more refined Pareto front.
Key Results
- Faster Convergence: On PyMOO tasks, HPI-ParEGO began outperforming the standard ParEGO baseline after only ~30% of the total trial budget.
- Superior Pareto Fronts: In the
LCBenchtasks, HPI-ParEGO consistently achieved lower "Hypervolume Regret" (a measure of how close the found solutions are to the true optimal trade-offs) than ParEGO, NSGA-II, and MO-TPE. - Robustness: While MO-TPE performed well in the early stages of some tasks, HPI-ParEGO was more robust across different types of benchmarks, eventually overtaking other methods in the final stages of the search.
Limitations and Future Work
- Surrogate Reliability: The HPI calculation depends on the accuracy of the surrogate model. In the very early stages of optimization (low data), HPI estimates might be noisy.
- Computational Overhead: Calculating Shapley values adds a small computational cost to each iteration. While this is usually negligible compared to the cost of training an ML model, it could be a factor for extremely cheap-to-evaluate functions.
- High-Dimensional Scaling: The authors note that for pipelines with hundreds of hyperparameters (like Auto-Sklearn), the overhead of HyperSHAP might become a bottleneck.
- Future Directions: The authors suggest exploring "meta-learning" to bring in HPI priors from previous tasks or using multi-fidelity optimization (e.g., training on small subsets of data) to speed up the HPI learning phase even further.
Summary updated