Imagine trying to build the ultimate racing vehicle. To save time, you decide on a two-step process: first, you test a dozen different chassis shapes using a standard, off-the-shelf engine setup. Once you find the fastest chassis, you swap in a high-performance turbo engine and spend days fine-tuning its fuel injection and gear ratios.
This sounds logical, but it contains a fatal flaw: the chassis that performed best with the basic engine might be completely unsuited for the turbo engine. The optimal aerodynamics, weight distribution, and suspension stiffness are deeply dependent on the power and behavior of the engine itself.
For years, automated machine learning (AutoML) made a similar mistake. It treated Neural Architecture Search (NAS)—finding the right arrangement of layers and connections—and Hyperparameter Optimization (HPO)—tuning learning rates, batch sizes, and regularization—as entirely separate problems.
Early automated design frameworks, such as the one proposed by Bowen Baker et al., evaluated candidate network topologies using fixed, constant training hyperparameters. This decoupled approach assumes that an architecture's intrinsic value can be measured independently of how it is trained. However, a growing body of research shows that structure and training configurations are fundamentally coupled. To find the true performance frontier of deep learning models, we must optimize both simultaneously.
The Hidden Coupling
Why does decoupling fail? The core issue is that the relative performance ranking of neural architectures is highly sensitive to the training recipe.
If you evaluate three different neural network architectures ($A$, $B$, and $C$) using a default learning rate and data augmentation strategy, architecture $A$ might come out on top. But if you change the learning rate or increase the intensity of the data augmentation, architecture $C$ might suddenly outperform $A$ by a wide margin.
As demonstrated by Xiaoliang Dai et al. in their work on FBNetV3, an architecture that appears suboptimal under one training recipe can become the state-of-the-art choice when paired with its ideal hyperparameters. Conversely, Archit Bansal et al. demonstrate in JAHS-Bench-201 that a simpler, structurally "inferior" architecture paired with well-tuned hyperparameters frequently beats a superior architecture saddled with a poor training recipe.
Decoupled Search (Suboptimal):
[Step 1: NAS] ---> Finds "Best" Architecture under Default HPO
[Step 2: HPO] ---> Tunes Hyperparameters *only* for that single Architecture
Joint Search (Optimal):
[Unified Space] ---> Simultaneously explores (Architecture × Hyperparameters)
---> Discovers synergistic pairings missed by sequential search
When we optimize architecture and hyperparameters separately, we do not just miss out on minor performance gains; we actively misjudge the quality of the architectures we are evaluating.
The Unified Search Space
Joint Architecture and Hyperparameter Search (JAHS) fixes this by merging discrete structural choices and continuous training parameters into a single, combinatorial search space.
Instead of asking "What is the best architecture?" followed by "What are the best hyperparameters?", a joint framework asks: "What is the best end-to-end training pipeline?"
This holistic framing allows the optimization algorithm to discover unexpected synergies between model structure and training routines. For instance, Lucas Zimmer et al. developed Auto-PyTorch to jointly search for network structures and training configurations on tabular data, proving that their interaction is too significant to ignore. Taking this a step further, Dong Yang et al. introduced T-AutoML, a framework that expands the searchable space even further to include data augmentation strategies alongside architecture and hyperparameters. By treating the entire pipeline as a unified entity, the system ensures that the model's structural capacity perfectly matches the complexity of the data transformations it encounters during training.
Navigating the Heterogeneous Landscape
While a unified search space solves the coupling problem, it introduces a massive computational challenge. The search space becomes highly heterogeneous, mixing:
- Discrete variables: Number of layers, operation types (e.g., convolution vs. self-attention).
- Continuous variables: Learning rates, weight decay coefficients.
- Permutation/Categorical variables: Data augmentation order or optimizer types.
To navigate this complex landscape without consuming prohibitive amounts of compute, researchers rely on several key algorithmic paradigms:
Multi-Fidelity and Meta-Learning
Instead of training every candidate configuration from scratch to full convergence, modern frameworks use cheap approximations (low-fidelity evaluations) to eliminate poor configurations early. To accelerate this, the system can look at past tasks to warm-start the search. For example, Sebastian Pineda Arango et al. developed Quick-Tune, which uses a meta-learned accuracy predictor and cost-aware Bayesian optimization to simultaneously select a pretrained model architecture and its optimal finetuning hyperparameters in a fraction of the usual time.
Differentiable Joint Search
In weight-sharing and differentiable architecture search, the coupling between structure and optimization is mathematical. The authors of IDARTS demonstrate that operation selection and connection weights are intrinsically linked; the perceived importance of a specific neural operation depends entirely on the strength of the edge it occupies. By treating this as a bilinear optimization problem and coordinating the updates of both structural and routing parameters, they prevent the search from collapsing into unstable local minima.
Broader Dimensions of Joint Optimization
The realization that system components cannot be optimized in isolation has spread far beyond standard image classification and tabular models. The same joint optimization philosophy is being applied to solve multi-objective trade-offs and entirely different paradigms:
- Fairness and Bias Reduction: Rhea Sanjay Sukthanker et al. and Yi Sheng et al. have shown that algorithmic bias in face recognition and medical imaging is deeply tied to the interaction between network structure and training loss formulations. By co-optimizing data sampling, loss weights, and network architectures, they discover models that balance accuracy and demographic fairness.
- Reinforcement Learning: Julian Dierkes et al. identify a "circular dependency" between an RL agent's hyperparameters and its reward shaping functions. Optimizing them together allows the agent to find global optima that independent tuning misses entirely.
- Large Language Models and Prompts: Even in the era of frozen or foundational models, the joint paradigm persists. Claudio Spiess applies this framing to prompt engineering, demonstrating that an LLM agent's reasoning structure (the architecture) and its system instructions (the hyperparameters) must be co-optimized to achieve maximum task accuracy.
By tearing down the artificial walls between how a model is shaped and how it is trained, joint optimization frameworks align the search process with the reality of deep learning: everything impacts everything else.