Neural Architecture Search (NAS) aims to automate the design of neural networks, replacing human intuition with algorithmic optimization. Early approaches relied on reinforcement learning or evolutionary algorithms, which required training thousands of candidate models from scratch—a process consuming massive computational budgets.
Differentiable Neural Architecture Search (DARTS) revolutionized the field by framing architecture design as a continuous optimization problem. Instead of picking a single operation (like a $3\times3$ convolution or a max-pooling layer) at each step, differentiable NAS constructs a massive supernetwork (or supernet) where every possible operation coexists in a parallel, weighted combination. This allows the algorithm to use gradient descent to jointly optimize both the network's weights and its structural parameters.
However, this elegance comes with a severe physical limitation: the supernetwork memory explosion.
The Mechanics of the Memory Explosion
To understand why differentiable NAS is so memory-intensive, we must look at how deep learning frameworks handle backpropagation.
In a standard neural network layer, an input tensor $X$ passes through a single operation $f(X)$ to produce an output $Y$. During the forward pass, the framework stores the intermediate activations of $f(X)$ in GPU memory so they can be used to compute gradients during the backward pass.
Standard Layer:
[Input X] ---> [ 3x3 Conv ] ---> [Output Y] (Stores 1 set of activations)
In a differentiable supernetwork, an edge between two network nodes does not contain just one operation; it contains an entire bank of $N$ candidate operations. The output of the edge is a weighted sum of all candidates:
$$Y = \sum_{i=1}^{N} \alpha_i f_i(X)$$
where $\alpha_i$ represents the architectural weight of operation $f_i$. Because the framework must compute gradients for every single candidate operation with respect to $\alpha$, the system must execute and store the activations for all $N$ operations simultaneously in GPU memory.
Supernet Layer:
+---> [ 3x3 Conv ] ---+
| |
[Input X] -----+---> [ 5x5 Conv ] ---+---> [Weighted Sum] ---> [Output Y]
| |
+---> [ Max Pool ] ---+
(Stores activations for ALL branches concurrently)
As noted by Zheng et al., this multi-path evaluation forces a strict trade-off between the complexity of the search space and the scale of the models. If your search space includes 8 candidate operations per edge, your memory footprint during the search phase scales linearly with that number, quickly causing Out-Of-Memory (OOM) errors on standard hardware.
The Cost of Memory Constraints: The Proxy Trap
Because a full-scale supernetwork cannot fit into standard GPU memory, early differentiable frameworks were forced to use restrictive workarounds, often referred to as proxy tasks:
- Shallow Proxies: Searching on a network with only a fraction of the layers required for final deployment.
- Data Proxies: Searching on downscaled, lower-resolution datasets (like CIFAR-10) instead of the target large-scale dataset (like ImageNet).
As Xie et al. observe, hardware memory constraints fundamentally restrict the depth of the network during the search phase. This creates a severe "depth gap" or "search-evaluation gap": a structural pattern that performs well in a shallow, 5-layer proxy network often fails to capture the long-range dependencies and hierarchical features that emerge when the network is scaled up to 20 or more layers for final evaluation.
To make automated architecture design truly practical, researchers had to find ways to run gradient-based search without instantiating the entire supernetwork in memory at once. Three core strategies have emerged to break this bottleneck.
Strategy 1: Path Binarization and Stochastic Sampling
If the memory explosion is caused by executing all parallel branches simultaneously, the most direct solution is to only execute a fraction of them during any given forward pass.
Cai et al. introduced this concept via path binarization in ProxylessNAS. Instead of keeping all candidate operations active, the framework uses the architecture parameters to define a probability distribution over the paths. During the forward pass, only a single path is sampled and instantiated in memory based on these probabilities.
Binarized Forward Pass:
+---> [ 3x3 Conv ] (Inactive)
|
[Input X] -----+---> [ 5x5 Conv ] (Active) ---> [Output Y]
|
+---> [ Max Pool ] (Inactive)
(Memory footprint equals a single standard network)
To update the architecture parameters using gradients, the authors sample a second path and update their relative weights using a binary gating mechanism. This bounds the memory overhead to that of a standard, single-path network, allowing researchers to bypass proxy tasks entirely and search directly on large-scale target datasets.
Strategy 2: Progressive Growth and Space Pruning
Another approach mitigates memory overhead by dynamically altering the size of the search space over time, ensuring the supernetwork is never fully dense and deep at the same moment.
Xie et al. proposed a progressive learning scheme in P-DARTS. The search begins with a shallow supernetwork where the memory footprint is low enough to accommodate all candidate operations. As training progresses, unpromising operations are iteratively pruned from the search space based on their architectural weights. As the number of candidates per edge drops, the saved memory is repurposed to increase the depth of the supernetwork. This gradually bridges the depth gap without exceeding hardware memory limits.
Similarly, Yan et al. developed RADARS, which treats memory capacity as a hard constraint. It uses a reinforcement learning controller to prune massive search spaces into manageable subspaces before constructing the differentiable supernetwork, preventing exponential memory growth from occurring in the first place.
Strategy 3: Weight Entanglement and Superkernels
A third paradigm shifts the focus from how we sample the search space to how we define the operations themselves. Traditional NAS treats a $3\times3$ convolution and a $5\times5$ convolution as two completely independent operations running on parallel branches.
Stamoulis et al. challenged this view by introducing Single-Path NAS. They noted that a $3\times3$ convolution kernel can be viewed as the inner core of a $5\times5$ convolution kernel. Instead of instantiating two separate branches, they construct a single superkernel containing the largest possible filter size.
Weight-Entangled Superkernel (5x5):
[ W_11 W_12 W_13 W_14 W_15 ]
[ W_21 [ w_11 w_12 w_13 ] W_25 ]
[ W_31 [ w_21 w_22 w_23 ] W_35 ] <-- Inner 3x3 weights shared
[ W_41 [ w_31 w_32 w_33 ] W_45 ] with outer 5x5 weights
[ W_51 W_52 W_53 W_54 W_55 ]
To evaluate a $3\times3$ convolution, the network simply applies a mask that zeroes out the outer weights. As Wang et al. demonstrate in MergeNAS, merging distinct operations into a single weight-sharing structure collapses the multi-path supernetwork into a single linear path.
Sukthanker expands on this concept by combining weight entanglement with gradient-based search via weight superposition. This eliminates the linear memory scaling typically caused by storing independent activations for every candidate operation, enabling gradient-based search to scale to massive macro-architectures that were previously completely inaccessible.
Summary
The evolution of memory-constrained architecture search reflects a shift away from brute-force supernetwork evaluation toward memory-aware design. Whether through stochastic path sampling (Cai et al.), progressive search space pruning (Xie et al.), or structural weight sharing (Stamoulis et al.), managing the activation memory footprint is what ultimately allows automated network design to scale from toy problems to real-world deployment.
Key Foundational Work
V-NAS: Neural Architecture Search for Volumetric Medical Image Segmentation (2019) — contributed 2 concept elements to this direction.
Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild (2019) — contributed 2 concept elements to this direction.
Multinomial Distribution Learning for Effective Neural Architecture Search. (2019) — contributed 2 concept elements to this direction.