Optimizing Conditional and Hierarchical Parameter Spaces

58 concepts45 papers2 proposals

When optimizing the performance of a complex system—whether it is a deep neural network, an automated machine learning (AutoML) pipeline, or an operating system kernel—we rarely face a simple, flat vector of independent numbers. Instead, choices are deeply nested.

Imagine configuring an AutoML pipeline. Your first decision might be choosing between a Support Vector Machine (SVM) and a Random Forest. If you choose the SVM, you must then tune its regularization constant $C$ and kernel coefficient $\gamma$. If you choose the Random Forest, $C$ and $\gamma$ become completely irrelevant, and you must instead tune the maximum tree depth and splitting criterion.

This is the core of a conditional or hierarchical search space. Parameters do not exist in a vacuum; their relevance, and sometimes their very existence, is contingent upon upstream decisions. Standard optimization frameworks often struggle with these conditional spaces, as noted by Jasper Snoek et al. in their foundational work on Bayesian optimization (Practical Bayesian Optimization of Machine Learning Algorithms). When we force a hierarchical problem into a flat optimization framework, we break the underlying geometry that optimization algorithms rely on to learn.

The Failure Mode of Flattening

The simplest way to handle conditional parameters is to ignore their structure. We can "flatten" the hierarchy into a single fixed-length vector containing every possible parameter across all branches. If a parameter is inactive for a given choice, we can pass a dummy value or a placeholder.

However, as Marcel Wever et al. demonstrate, flattening introduces severe inefficiencies (AutoML for Multi-Label Classification: Overview and Empirical Evaluation). It creates two major mathematical hurdles:

  1. The Distance Distortion: Standard black-box optimizers, such as Gaussian Processes (GPs), rely on calculating distances between points in the search space to predict performance. If two configurations both select a Random Forest but contain wildly different dummy values for the inactive SVM parameter $\gamma$, a standard flat kernel will treat these two configurations as distant points. In reality, their performance will be identical because $\gamma$ is inactive. This injects massive amounts of noise into the surrogate model.
  2. Combinatorial Explosion: In large-scale systems, the number of inactive parameters can dwarf the active ones. For example, Huilai Chen et al. model the 15,000+ configuration options of the Linux kernel as a dynamic tree, where a single high-level toggle can reveal or hide hundreds of sub-options (AutoOS: Make Your OS More Powerful by Exploiting Large Language Models). Flattening such a space creates an unmanageable dimensionality where the optimizer wastes precious evaluations sampling irrelevant parameters, a challenge also highlighted in benchmark suites like HPOBench by Katharina Eggensperger et al. (HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO).

To solve this, researchers have developed optimization paradigms that natively understand trees, graphs, and conditional routing.

Inverting the Model: Tree-Structured Density Estimation

One of the earliest breakthroughs in navigating hierarchical spaces came from changing what the surrogate model tries to predict. Instead of modeling the objective function directly as a function of the parameters ($P(\text{performance} \mid \text{parameters})$), we can invert the probability using Bayes' rule to model the distribution of parameters given performance ($P(\text{parameters} \mid \text{performance})$).

This is the core mechanism behind the Tree-Structured Parzen Estimator (TPE), introduced by James Bergstra et al. (Algorithms for Hyper-Parameter Optimization). TPE splits past evaluations into two groups: "good" performing configurations and "bad" performing configurations. It then fits a separate probability density to each group.

          [Algorithm Selection]
             /             \
       (SVM Chosen)   (Random Forest Chosen)
           /                 \
     P(C, γ | Good)     P(Depth | Good)

Because the density estimator is structured as a tree, it naturally mirrors the conditional logic of the search space. When calculating the probability of a configuration, the algorithm applies the chain rule along the active branch:

$$P(\text{algorithm}, \text{param} \mid \text{performance}) = P(\text{algorithm} \mid \text{performance}) \cdot P(\text{param} \mid \text{algorithm}, \text{performance})$$

If the SVM branch is not selected, the parameters specific to the SVM are completely excluded from the likelihood calculation for that sample. This allows algorithms like TPE to efficiently scale to hundreds of dimensions in complex vision architectures and multi-objective pipelines, as shown by Dan Yamins (Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures) and Shuhei Watanabe et al. (Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator).

Fixing the Geometry: Conditional Kernels and Dynamic Graphs

If we want to retain the benefits of Gaussian Processes—such as robust uncertainty estimation—we must modify the kernel function so that it respects the tree structure.

Julien-Charles Lévesque et al. proposed specialized Conditional Kernels to fix distance calculations (Bayesian optimization for conditional hyperparameter spaces). Instead of computing a standard Euclidean distance across all dimensions, a conditional kernel checks whether two configurations share the same active parameters. If two configurations diverge at a high-level node (e.g., one chooses an SVM and the other a Random Forest), the kernel forces their structural distance to a baseline maximum, ignoring the values of their downstream parameters. If they share the same branch, the kernel only computes distances across the parameters that are active for that specific branch.

An alternative to hardcoding these hierarchies upfront is to let the search space construct itself during runtime. Takuya Akiba et al. introduced this concept via a "define-by-run" API in Optuna (Optuna: A Next-generation Hyperparameter Optimization Framework). Instead of declaring a static configuration space before the optimization begins, the search space is implicitly defined by the execution path of the code itself. As the optimization program runs, conditional loops and branches dynamically register parameters only when they are encountered, allowing the internal relational tracking to infer dependencies on the fly.

Decomposing Complexity via Sequential Decisions

Another powerful intuition is to treat hierarchical optimization not as a single, massive choice, but as a multi-step game or sequential decision process.

For instance, the Combined Algorithm Selection and Hyperparameter optimization (CASH) problem can be decomposed into a two-layer process. As implemented in the FLASH framework by Yuyu Zhang et al., a fast parametric model first prunes the high-level algorithm space, and then a secondary optimization layer tunes the hyperparameters of the surviving candidates (FLASH: Fast Bayesian Optimization for Data Analytic Pipelines). This division of labor prevents unified optimizers from choking on the joint search space complexity (Amir Rezaei Balef et al. (Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learning)).

This sequential framing maps naturally to tree-search and reinforcement learning algorithms:

  • Monte-Carlo Tree Search (MCTS): Herilalaina Rakotoarison et al. treat machine learning pipeline construction as a path-finding problem on a tree (Automated Machine Learning with Monte-Carlo Tree Search). MCTS navigates the discrete choices of pipeline structure (e.g., preprocessing $\rightarrow$ feature selection $\rightarrow$ classifier), while a standard Bayesian optimizer handles the continuous hyperparameters activated at the leaf nodes.
  • Multi-Agent Ordering: Chen Lu et al. formulated conditional optimization as a cooperative multi-agent game using a Sequential Advantage Decomposition Network (Sequential Multi-Agent Dynamic Algorithm Configuration). By enforcing a strict logical order of execution, upstream agents make high-level categorical choices, allowing downstream agents to specialize their tuning strategies entirely within the active parameter bounds.
[Upstream Agent] ---> Makes High-Level Choice (e.g., Algorithm Type)
                             |
                             v
[Downstream Agent] --> Specializes within the activated sub-space

This bi-level decomposition drastically reduces the effective action space size, ensuring that the optimization process never wastes evaluations on illegal or nonsensical parameter combinations (Thomy Phan et al. (Adaptive Anytime Multi-Agent Path Finding Using Bandit-Based Large Neighborhood Search)).

Broader Horizons

The shift from flat vector optimization to hierarchical modeling has unlocked optimization capabilities across diverse domains. It allows for the formalization of programmable, modular neural architecture searches using lazy substitution nodes (Renato Negrinho (Towards modular and programmable architecture search)) and layer-by-layer conditional design choices (Aliasghar Mortazi (Automatically Designing CNN Architectures for Medical Image Segmentation)). It also prevents resource waste when tuning complex genetic algorithms where crossover parameters are structurally dependent on population thresholds (Furong Ye et al. (Automated Configuration of Genetic Algorithms by Tuning for Anytime Performance)).

While modern sequence-based models like OptFormer (Yutian Chen et al. (Towards Learning Universal Hyperparameter Optimizers with Transformers)) still face challenges in natively processing complex conditional logic, explicitly embedding structural dependencies into search spaces remains a vital strategy for mastering high-dimensional, non-Euclidean optimization landscapes (Xingchen Wan et al. (Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces)).

Key Foundational Work

Algorithms for Hyper-Parameter Optimization (2011) — contributed 3 concept elements to this direction.
Making a Science of Model Search (2012) — contributed 1 concept elements to this direction.
Practical Bayesian Optimization of Machine Learning Algorithms (2012) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 45 papers
  2. Version 1 · 2026-10-04 · 91 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_search_spaces_23e90a4b,
  title = {hierarchical search spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-search-spaces/rdr_23e90a4b45e34408b48bc137442513d5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_hierarchical_search_spaces_23e90a4b_v2,
  title = {hierarchical search spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-search-spaces/rdr_23e90a4b45e34408b48bc137442513d5/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake