Early Neural Architecture Search (NAS) frameworks were computationally ruinous, often requiring thousands of GPU days to discover a single high-performing model. To make automated design practical, early breakthrough methods introduced a clever structural constraint: instead of searching for the entire network topology from scratch, the algorithm optimizes a small, modular motif called a cell. Once this cell—typically a small Directed Acyclic Graph (DAG)—is optimized, it is replicated and stacked sequentially according to a fixed, human-designed macro-skeleton.
As noted by Yuhui Xu et al. in PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search, this cell-based paradigm drastically reduces the search space, making gradient-based optimization tractable. However, it acts as a structural cage. It forces every layer of the network to share the exact same local topology, completely preventing the discovery of heterogeneous macro-architectures where early layers might need entirely different connectivity patterns than deeper layers. Moving beyond these rigid templates to unconstrained macro-architectures is the next frontier in automated model design.
The Combinatorial Explosion of Global Search
Why can't we simply remove the cell constraint and search the entire network graph end-to-end? The fundamental barrier is the space explosion problem. If a cell has 4 internal nodes, the number of possible internal topologies is small and easily navigable. But if a network has 20 layers and any layer can connect to any other layer, the number of unique global graphs explodes exponentially.
Yaofo Chen et al. demonstrate in Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search that when a search space becomes this massive, random sampling or naive gradient updates provide almost zero useful guidance. The searcher becomes lost in an ocean of invalid or poorly performing architectures. To make unconstrained macro-architecture search work, algorithms must find ways to structure or factorize this massive global space without re-imposing rigid modular repetition.
Strategies for Unconstrained Macro Search
To break free from the cell template, modern frameworks employ distinct algorithmic mechanisms designed to make global topology search stable and computationally efficient.
Curriculum and Progressive Growth
Instead of optimizing a massive, deep global graph from day one, we can smoothly scale the complexity of the search space over time. Yaofo Chen et al. propose starting with a tiny, simple search space and gradually introducing more complex operations and paths as the searcher learns.
Similarly, Lingxi Xie et al. show in Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation that gradually increasing the depth of the search network allows the algorithm to handle the structural shifts that occur when moving from shallow proxy tasks to deep target networks, preventing optimization collapse in deep macro-structures.
[Phase 1: Shallow Backbone] ---> [Phase 2: Add Intermediate Layers] ---> [Phase 3: Full Macro Graph]
(Optimize basic paths) (Introduce complex routing) (Fine-tune global topology)
Disentangling Topology from Operations
In standard differentiable NAS, an algorithm simultaneously decides which operation to place on an edge (e.g., a $3\times3$ convolution vs. a pooling layer) and whether that edge should exist at all. In a macro search space, this dual optimization often leads to "performance collapse," where the network becomes flooded with parameter-free skip connections because they propagate gradients faster early in training.
To fix this, Xiangxiang Chu et al. introduce a method called ROME (ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradient Accumulation) that explicitly decouples the search for the global network backbone from the selection of individual operations, stabilizing the optimization of irregular topologies.
Hierarchical and Tree-Based Factorization
Another way to tame the complexity of global graphs is to factorize the search space into hierarchical decisions. Guocheng Qian et al. propose TNAS (When NAS Meets Trees: An Efficient Algorithm for Neural Architecture Search), which splits the search space into two distinct tree structures: one managing the layer-level macro-skeleton and the other managing operation selection. By using a bi-level breadth-first search to prune unpromising branches of these trees, they find optimal global structures efficiently.
Alternatively, Benteng Ma et al. model the macro-structure by learning explicit transition matrices between topologically connected layers (Inter-layer transition in neural architecture search), capturing the structural flow of information across the entire network rather than treating layers as independent entities.
Mapping Irregular Topologies to Continuous Spaces
When we abandon cell-based templates, we also lose the ability to easily encode architectures as simple fixed-length strings or discrete tokens. An unconstrained macro-architecture is an arbitrary graph, and predicting its performance requires a representation that can capture complex, non-modular structural variations.
To solve this, researchers leverage graph neural networks to learn continuous embeddings of neural topologies. Shen Yan et al. introduce arch2vec (Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?), which uses an unsupervised Variational Graph Isomorphism Autoencoder to reconstruct network architectures without relying on performance labels. This creates a smooth, continuous latent space where structurally similar global topologies sit close to one another, making downstream search significantly more sample-efficient.
Building on this idea, Jovita Lukasik et al. demonstrate in Smooth Variational Graph Embeddings for Efficient Neural Architecture Search that smooth graph embeddings allow performance predictors to generalize effectively even to architectures that are larger or more complex than those seen during training, breaking the structural boundaries of traditional fixed-space predictors.
Discrete Macro Graphs Continuous Latent Space Performance Prediction
[Graph A (20 layers)] \ /---> Predictor -> High Score
---> [ Graph Autoencoder ] ---> ( Smooth Space )
[Graph B (24 layers)] / \---> Predictor -> Low Score
Why Global Macro Search Matters
The constraints of cell-based repetition inherently limit a model's ability to adapt to complex real-world trade-offs. For instance, in specialized domains like designing models with strict privacy guarantees, the authors of DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy point out that rigid cell structures limit the discovery of non-repetitive global configurations that could offer superior privacy-utility tradeoffs.
By transitioning from localized cell optimization to unconstrained macro-architectural design, the field moves closer to truly automated, hardware-aware, and task-optimized neural network discovery, unlocking structural designs that human intuition would never have composed.