Generalizing Neural Architecture Search Across Structural Paradigms

47 concepts44 papers1 proposal

Neural Architecture Search (NAS) promises to automate the design of optimal deep learning models, removing human bias from network engineering. To make this computationally tractable, researchers define a search space—a bounded sandbox of possible configurations. Historically, these sandboxes have been highly constrained and homogeneous: cell-based convolutional networks or mobile-scale inverted residual blocks.

Within these narrow sandboxes, efficient performance estimators (or "proxies") work remarkably well. Instead of training thousands of candidate networks to convergence, NAS algorithms use shortcuts like zero-shot indicators, weight-sharing supernets, or random-weight evaluations to rank candidates in seconds.

However, a fundamental crisis has emerged in automated machine learning: these proxies do not generalize across heterogeneous structural paradigms. A shortcut that perfectly ranks convolutional cells frequently falls apart when asked to evaluate a Vision Transformer (ViT), a state-space model, or an unconventional hybrid architecture. The field is hitting a wall where search spaces and evaluation metrics cannot easily transfer or scale beyond the specific inductive biases they were engineered for.

Why Performance Proxies Fail to Cross Borders

To understand why search spaces lack transferability, we must look at how performance proxies operate. Consider "zero-cost" proxies, which evaluate a network at initialization without any training. For instance, Joe Mellor et al. introduced Neural Architecture Search without Training (NASWOT), which uses the overlap of linear activation patterns to rank networks. While highly effective for convolutional networks, the authors note that it remains an open question whether these activation-based metrics generalize to attention-based Transformers or non-vision domains.

Similarly, Random-Weight Evaluation (RWE)—explored by Shengran Hu et al. in Multi-objective Neural Architecture Search with Almost No Training and Accelerating multi-objective neural architecture search by random-weight evaluation—estimates a network's potential by freezing its backbone with random weights and training only the final classification layer. While this cuts search time to minutes within convolutional cell-based spaces, its ranking signals lose accuracy when applied to fundamentally different structural paradigms like Transformers.

The core issue is that these proxies are highly sensitive to specific topological constraints. As Xuefei Ning et al. demonstrate in Evaluating Efficient Performance Estimators of Neural Architectures, zero-shot estimators exhibit massive performance variance across different search spaces due to inherent structural biases, such as favoring larger models or penalizing skip connections. Because there is no universal proxy, a ranking signal optimized for one family of neural topologies fails to generalize to another.

The Breakdown of Weight-Sharing and Structural Decoupling

Another popular shortcut is the "one-shot" supernet, where all candidate architectures share a single set of weights to avoid training from scratch. In K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets, Xiu Su et al. point out that while multi-weight supernets mitigate multi-model forgetting in convolutional spaces, their generalizability to irregular connectivity patterns or Transformer blocks remains unproven.

When search spaces attempt to decouple network structure from specific operations to improve efficiency—as explored by Jinu Lee et al. in Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling—the decoupling mechanisms themselves prove paradigm-dependent. A strategy that separates structure from operations in a convolutional cell fails to translate to Graph Neural Networks or Transformers, where connectivity and attention mechanisms are fundamentally intertwined.

Even meta-learning approaches designed to train unbiased supernets, such as the framework introduced by Hyeonmin Ha et al. in SUMNAS: Supernet with Unbiased Meta-Features for Neural Architecture Search, have only been validated on homogeneous search spaces. When a search space becomes highly diverse—mixing fundamentally different architectures like CNNs and Transformers—learning a single set of unbiased meta-features becomes an unresolved challenge.

Overfitted Robustness and Hardware Predictors

The generalizability crisis extends beyond accuracy estimation to multi-objective constraints like hardware latency, quantization, and adversarial robustness.

Consider hardware performance predictors. Amir Yazdanbakhsh demonstrates in An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks that while Graph Neural Networks can predict latency and energy with 97% accuracy within a specific convolutional search space, these learned models face extreme uncertainty when transferred to fundamentally different network classes like recurrent structures or generative adversarial networks.

Similarly, joint architecture and quantization frameworks (such as ∇QDARTS by Payman Behnam et al.) and information-based importance metrics (such as IS-DARTS by Hongyi He et al.) are heavily optimized for convolutional primitives. Their empirical scaling behavior on non-convolutional structures remains unverified.

Even security and robustness metrics are bound to their structural sandboxes. Zero-cost robustness proxies like CRoZe, developed by Hyeonjeong Ha et al. in Generalizable Lightweight Proxy for Robust NAS against Diverse Perturbations, evaluate internal structural consistency at initialization to find resilient models. Yet, the authors acknowledge these metrics lack proven transferability to non-cell-based designs like Transformers or MLP-Mixers.

This structural lock-in means that vulnerabilities discovered in one search space—such as the structural backdoors identified by Ren Pang et al. in The Dark Side of AutoML: Towards Architectural Backdoor Search—may manifest entirely differently or fail to map onto large-scale attention models.

Bridging the Gap: Towards Universal Search Spaces

To overcome the limitations of paradigm-locked search spaces, recent research is exploring ways to build more flexible, cross-architecture discovery frameworks.

One approach is to leverage the broad, cross-domain intuition of Large Language Models. Ganesh Jawahar et al. demonstrate in LLM Performance Predictors are good initializers for Architecture Search that LLMs can predict architecture performance across hyperparameters, though their evaluations remain largely confined to Transformer families. Taking this a step further, Xingyu Wu introduces Design Principle Transfer in Neural Architecture Search via Large Language Models (LAPT), which uses LLMs to extract linguistic "design principles" from successful networks to prune massive search spaces into optimized subspaces. However, even these automated refinement techniques face validation hurdles when moving from cell-based CNNs to Graph Neural Networks.

Ultimately, the goal is to discover universal scaling laws and indicators that govern performance regardless of inductive bias. As Xiaohua Zhai et al. show in Scaling Vision Transformers, scaling laws derived for one architecture (like ViTs) do not automatically apply to models with different inductive biases. True search space transferability will require moving away from brittle, topology-specific proxies and toward fundamental theoretical indicators of network trainability and expressivity that remain invariant across the entire spectrum of neural architectures.

Key Foundational Work

Multi-objective Neural Architecture Search with Almost No Training (2020) — contributed 1 concept elements to this direction.
Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS (2020) — contributed 1 concept elements to this direction.
Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_search_space_transferability_797314c5,
  title = {search space transferability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/search-space-transferability/rdr_797314c5b1904b59b8aa42905c3c43ba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_search_space_transferability_797314c5_v1,
  title = {search space transferability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/search-space-transferability/rdr_797314c5b1904b59b8aa42905c3c43ba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake