MergeNAS: Merge Operations into One for Differentiable Architecture Search

Xiaoxing Wang, Chao Xue, Junchi Yan, Xiaokang Yang, Yonggang Hu, Kewei Sun

Abstract

Differentiable Architecture Search (DARTS) has transformed Neural Architecture Search (NAS) by framing it as a continuous optimization problem. However, DARTS suffers from significant memory overhead due to maintaining all candidate operations simultaneously and instability caused by the dominance of parameterless operations. We present MergeNAS, a novel "one-shot" NAS method that merges multiple candidate operations into a single operation with shared weights. This approach reduces memory bloat and prevents overfitting by treating various operations as masked versions of one large kernel. MergeNAS achieves state-of-the-art accuracy on benchmarks like NAS-Bench-201 in as little as 0.2 GPU-days.

Introduction

DARTS revolutionized the field but introduced two major challenges that hinder its scalability and reliability. First, Memory Bloat: searching for the optimal operation on an edge requires keeping all candidates (e.g., $3 \times 3$ conv, $5 \times 5$ conv, max pooling) in memory simultaneously, which limits the ability to search on large models or high-resolution data. Second, Performance Collapse: the search often overfits to parameterless operations like skip-connections early in the process, leading to architectures that perform poorly on test data despite high training performance.

MergeNAS addresses these by reducing search space redundancy through a technique called weight merging.

Methodology

The core innovation of MergeNAS is Weight Merging. Instead of treating different-sized or dilated convolutions as separate entities, they are represented as a single "master" kernel.

Illustration of merging various convolutional operations into a single mask matrix $\tilde{M}$.

Mathematically, a smaller kernel can be viewed as a larger kernel with zero-padding, and a dilated convolution is simply a standard convolution with zeros inserted between weights. By maintaining one large master kernel and applying a "saliency matrix" (a weighted mask), MergeNAS can represent the weighted average of all candidate operations—the standard DARTS approach—within a single operation. This logic extends to separable convolutions and even parameterless operations like identity mappings, which can be represented as a convolution where the center weight is 1 and all others are 0.

The algorithm trains only this single merged operation per edge during the search phase, drastically reducing the parameter count and memory footprint compared to traditional one-shot methods.

Experimental Setup

MergeNAS was evaluated on several standard computer vision benchmarks, including CIFAR-10, CIFAR-100, and ImageNet-16-120. Crucially, the authors utilized NAS-Bench-201, a benchmark that provides ground-truth performance for every possible architecture in a fixed search space. This allows for a direct assessment of whether the search algorithm actually converges to the optimal solution rather than just a local minimum.

Results and Discussion

  • Efficiency: MergeNAS is exceptionally fast, completing a CIFAR-10 search in 0.2 to 0.6 GPU-days on a single NVIDIA 1080 Ti.
  • Performance: On CIFAR-10, it achieved a test error of 2.68% with a model size of approximately 2.9M parameters, outperforming standard DARTS (2.76%) and other popular variants like SNAS and P-DARTS.
  • Stability: Unlike DARTS, which often collapses after 20 epochs on NAS-Bench-201 due to skip-connection dominance, MergeNAS remains stable and consistently finds near-optimal architectures across different random seeds.
  • Implicit Regularization: Merging weights acts as a form of regularization. Shared weights force the model to learn features useful across different receptive fields simultaneously, preventing the model from "cheating" by over-relying on easy-to-optimize skip-connections.

Conclusion

MergeNAS demonstrates that redundancy is a significant bottleneck in current NAS frameworks. By merging operations, the method achieves massive reductions in computational cost while improving stability. While highly effective for convolutions and pooling, merging more complex operations like self-attention or non-linear transformations remains a challenge for future research. Nonetheless, this weight-merging strategy could be applied to other one-shot NAS frameworks to enable complex architecture searches on consumer-grade hardware.

Summary updated

Abstract

Differentiable architecture search (DARTS) has been a promising one-shot architecture search approach for its mathematical formulation and competitive results. However, besides its caused high memory utilization and a large computation requirement, many research works have shown that DARTS also often suffers notable over-fitting and thus does not work robustly for some new tasks. In this paper, we propose a one-shot neural architecture search method referred to as MergeNAS by merging different types of operations e.g. convolutions into one operation. This merge-based approach not only reduces the search cost (about half a GPU day), but also alleviates over-fitting by reducing the redundant parameters. Extensive experiments on different search space and various datasets have been conducted to verify our approach, showing that MergeNAS can converge to a stable architecture and achieve better performance with fewer parameters and search cost. For test accuracy and its stability, MergeNAS outperforms all NAS baseline methods implemented on NAS-Bench-201, including DARTS, ENAS, RS, BOHB, GDAS and hand-crafted ResNet.
BibTeX
@inproceedings{wang2020mergenas,
  author = {Xiaoxing Wang and Chao Xue and Junchi Yan and Xiaokang Yang and Yonggang Hu and Kewei Sun},
  title = {{MergeNAS: Merge Operations into One for Differentiable Architecture Search}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/424},
}
Figure