ZO-DARTS++: An Efficient and Size-Variable Zeroth-Order Neural Architecture Search Algorithm

Lunchen Xie, Eugenio Lomurno, Matteo Gambella, Danilo Ardagna, Manual Roveri, Matteo Matteucci, Qingjiang ShiarXiv 2025 · DOI · arXiv

Abstract

ZO-DARTS++ is a next-generation differentiable Neural Architecture Search (NAS) algorithm designed to accelerate model discovery while producing highly efficient, resource-constrained architectures. By integrating zeroth-order gradient approximation with a novel size-variable search scheme and sparsemax activation, it addresses the fundamental flaws of standard Differentiable Architecture Search (DARTS). Evaluated on the MedMNIST benchmark, ZO-DARTS++ achieves up to a 1.8% accuracy improvement over DARTS and reduces search time by nearly 40%, making it particularly suitable for medical imaging applications on edge devices.

Introduction

Designing deep learning architectures manually is a labor-intensive process requiring significant expertise. While Neural Architecture Search (NAS) was proposed to automate this, the popular DARTS framework suffers from three major limitations: optimization complexity due to intractable bi-level gradients, "operation collapse" where skip connections are unfairly favored over convolutions, and rigid resource management that requires separate searches for different model sizes. ZO-DARTS++ is introduced to overcome these hurdles, specifically targeting the demanding requirements of clinical medical imaging.

Methodology

The authors propose three primary technical innovations to refine the search process:

  • Zeroth-Order (ZO) Gradient Approximation: Instead of calculating computationally expensive Hessian matrices, ZO-DARTS++ estimates gradients by perturbing architecture parameters and observing loss changes. This approach is both more accurate and significantly faster than traditional first- or second-order approximations.
  • Sparsemax with Temperature Annealing: Unlike the standard softmax which maintains non-zero probabilities for all operations, sparsemax can assign exactly zero probability to underperforming operations. Combined with temperature annealing, this forces the model to converge toward a discrete, interpretable architecture.
  • Size-Variable Search Scheme: This "++" feature allows the algorithm to dynamically adjust cell depth and convolution kernel sizes. As shown in the architecture diagram below, smaller kernels inherit weights from the center of larger kernels, enabling the exploration of multiple model scales (Small, Medium, Large) in a single search pass without additional training costs.
Overview of the ZO-DARTS++ framework, including the searchable cell structure and the kernel-variable convolution mechanism with weight inheritance.

Experimental Results

The researchers evaluated ZO-DARTS++ on the MedMNIST benchmark, comprising 10 diverse 2D medical imaging datasets such as PathMNIST and OrganAMNIST.

Efficiency and Accuracy

ZO-DARTS++ proved to be 38.6% faster than the original DARTS and outperformed established baselines like ResNet-18 and Google AutoML. The "Large" variant achieved an average accuracy of 84.00%, a notable increase over the 82.20% achieved by standard DARTS. Furthermore, the use of sparsemax allowed for early stopping at 40 epochs, further reducing computational overhead.

Model Compactness

A key strength of the algorithm is its ability to produce resource-efficient models. The "Small" variant reduced parameter counts by over 35% compared to standard NAS models while maintaining competitive performance. The distribution of these parameters across the various MedMNIST datasets highlights the algorithm's ability to adapt model complexity to the specific task at hand.

Violin plots showing the distribution of model parameters (in millions) discovered by ZO-DARTS++ across the 10 MedMNIST datasets.

Conclusion

ZO-DARTS++ represents a significant step forward in making NAS practical for real-world applications. By combining gradient estimation efficiency with flexible model sizing, it provides a robust tool for discovering high-performance architectures for medical edge devices. Future work will focus on extending the framework to 3D medical data (e.g., CT/MRI scans) and automating the tuning of size-constraint hyperparameters.

Summary updated

Abstract

Differentiable Neural Architecture Search (NAS) provides a promising avenue for automating the complex design of deep learning (DL) models. However, current differentiable NAS methods often face constraints in efficiency, operation selection, and adaptability under varying resource limitations. We introduce ZO-DARTS++, a novel NAS method that effectively balances performance and resource constraints. By integrating a zeroth-order approximation for efficient gradient handling, employing a sparsemax function with temperature annealing for clearer and more interpretable architecture distributions, and adopting a size-variable search scheme for generating compact yet accurate architectures, ZO-DARTS++ establishes a new balance between model complexity and performance. In extensive tests on medical imaging datasets, ZO-DARTS++ improves the average accuracy by up to 1.8\% over standard DARTS-based methods and shortens search time by approximately 38.6\%. Additionally, its resource-constrained variants can reduce the number of parameters by more than 35\% while maintaining competitive accuracy levels. Thus, ZO-DARTS++ offers a versatile and efficient framework for generating high-quality, resource-aware DL models suitable for real-world medical applications.
BibTeX
@misc{xie2025zo,
  author = {Lunchen Xie and Eugenio Lomurno and Matteo Gambella and Danilo Ardagna and Manual Roveri and Matteo Matteucci and Qingjiang Shi},
  title = {{ZO-DARTS++: An Efficient and Size-Variable Zeroth-Order Neural Architecture Search Algorithm}},
  year = {2025},
  eprint = {2503.06092},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.06092},
}
Figure