TL;DR
Across-Task Neural Architecture Search (AT-NAS) combines gradient-based meta-learning with evolutionary algorithms to automatically find optimal neural network architectures for few-shot learning tasks. By decoupling weight optimization from architecture search, AT-NAS avoids the memory explosion typical of differentiable NAS methods, enabling the search for deeper networks with up to 60× less computational cost than searching from scratch.
What Problem Does This Paper Address?
Neural Architecture Search (NAS) has revolutionized deep learning by automating network design, but it remains notoriously resource-intensive, often requiring days or weeks of GPU compute. Applying NAS to multi-task or few-shot learning scenarios—where data is scarce and models must adapt rapidly to novel tasks—compounds this difficulty.
Prior attempts to merge meta-learning (like MAML) with gradient-based NAS (like DARTS) suffer from a major flaw: GPU memory explosion. Because DARTS-style methods continuously relax the discrete search space, they must keep all candidate operations in memory simultaneously during training. This limits the search space to shallow networks and small batches. The authors propose AT-NAS to break through this memory bottleneck, allowing researchers to search for deeper, high-performing architectures across a distribution of tasks without running out of VRAM.
Methods
AT-NAS introduces a hybrid approach that cleanly separates parameter optimization from architecture optimization:
- Weight-Sharing Supernet: Instead of keeping all candidate operations active simultaneously, AT-NAS trains a single "Supernet" where subnets are sampled discretely. Only one candidate operation per edge is selected at run-time, reducing GPU memory consumption to the level of a standard compact model.
- Meta Parameter Optimization: The Supernet's weights are optimized across multiple tasks using a first-order MAML/Reptile approximation, ensuring the shared weights provide a strong initialization for fast adaptation.
- Meta Architecture Optimization: Instead of using gradients to update architecture parameters, AT-NAS employs a Meta Continuous Evolutionary Algorithm (EA). Subnets are treated as individuals in a population, encoded as integer strings representing specific operations. High-performing architectures are preserved and mutated over generations based on their multi-task validation accuracy.
To make this pipeline practical, the authors introduce two key acceleration strategies:
- Distributed Parallel Meta-Search: Naive parallelization of MAML across multiple GPUs destabilizes Batch Normalization (BN) because few-shot tasks have tiny batch sizes. AT-NAS isolates the inner-loop optimization within independent processes on each GPU, accumulating gradients only after the inner steps are complete. This stabilizes BN and cuts search time linearly by the number of GPUs.
- Online Performance Predictor: To avoid the overhead of running validation inference on thousands of candidate subnets, the authors train a surrogate Random Forest regressor. It maps a subnet's architecture code directly to a predicted performance score, significantly speeding up the evolutionary selection step.
Datasets
The authors evaluated AT-NAS across two distinct paradigms:
- Few-Shot Learning Benchmarks:
- Omniglot: 1,623 handwritten characters from 50 alphabets, evaluated on 5-way and 20-way classification (both 1-shot and 5-shot).
- Mini-ImageNet: A standard few-shot benchmark consisting of 100 classes sampled from ImageNet, split into 64 training, 16 validation, and 20 testing classes.
- Supervised Learning Transfer Benchmarks:
- ImageNet-1K: Used to pre-train the meta-network.
- CIFAR-10 & SVHN: Target datasets used to evaluate how well the pre-trained meta-architecture transfers to standard, full-scale image classification tasks.
Key Insights and Ideas
- Decoupling is Key to Scale: By replacing the continuous relaxation of DARTS with an evolutionary algorithm over a discrete supernet, the authors decoupled architecture search from weight optimization. This dropped memory usage from over 10 GB to roughly 2 GB, making 8-layer deep searches feasible on standard hardware.
- Fixing Distributed MAML: The realization that naive data-parallel MAML breaks Batch Normalization due to small task sample sizes is a crucial engineering insight. Isolating the inner-loop optimization per GPU solves this instability.
- Ranking Over Absolute Accuracy: While surrogate performance predictors struggle to guess the exact accuracy of a network in few-shot settings, the authors found that a simple Random Forest model is excellent at ranking architectures (separating the good from the bad), which is all an evolutionary algorithm needs to succeed.
Key Results
- Memory Efficiency: AT-NAS required only 2044 ± 272 MB of GPU memory, compared to DARTS (10,050 MB) and PC-DARTS (3,235 MB), allowing the search of deeper network structures.
- Few-Shot Accuracy: On Mini-ImageNet (5-way, 5-shot), AT-NAS achieved a state-of-the-art accuracy of 75.56% (large model), significantly outperforming standard MAML++ (68.32%) and competing meta-NAS methods like T-NAS++ (69.59%).
- 60× Faster Transfer: When transferring the pre-trained meta-architecture to CIFAR-10 and SVHN, AT-NAS found optimal task-specific networks in less than an hour (0.05–0.06 GPU days). This matched or exceeded the accuracy of models searched from scratch while reducing the search cost by a factor of 60.
Limitations and Future Work
While the performance predictor successfully accelerates the evolutionary algorithm, the authors note that its absolute accuracy is relatively poor in few-shot scenarios compared to single-task scenarios with large datasets. This is because few-shot task distributions are highly volatile.
Additionally, the current framework is strictly evaluated on image classification. The authors explicitly state their intention to extend AT-NAS beyond classification to more complex multi-task domains like object detection or semantic segmentation in future work.
Summary updated