Abstract
ZO-DARTS++ is a next-generation differentiable Neural Architecture Search (NAS) algorithm designed to accelerate model discovery while producing highly efficient, resource-constrained architectures. By integrating zeroth-order gradient approximation with a novel size-variable search scheme and sparsemax activation, it addresses the fundamental flaws of standard Differentiable Architecture Search (DARTS). Evaluated on the MedMNIST benchmark, ZO-DARTS++ achieves up to a 1.8% accuracy improvement over DARTS and reduces search time by nearly 40%, making it particularly suitable for medical imaging applications on edge devices.
Introduction
Designing deep learning architectures manually is a labor-intensive process requiring significant expertise. While Neural Architecture Search (NAS) was proposed to automate this, the popular DARTS framework suffers from three major limitations: optimization complexity due to intractable bi-level gradients, "operation collapse" where skip connections are unfairly favored over convolutions, and rigid resource management that requires separate searches for different model sizes. ZO-DARTS++ is introduced to overcome these hurdles, specifically targeting the demanding requirements of clinical medical imaging.
Methodology
The authors propose three primary technical innovations to refine the search process:
- Zeroth-Order (ZO) Gradient Approximation: Instead of calculating computationally expensive Hessian matrices, ZO-DARTS++ estimates gradients by perturbing architecture parameters and observing loss changes. This approach is both more accurate and significantly faster than traditional first- or second-order approximations.
- Sparsemax with Temperature Annealing: Unlike the standard softmax which maintains non-zero probabilities for all operations, sparsemax can assign exactly zero probability to underperforming operations. Combined with temperature annealing, this forces the model to converge toward a discrete, interpretable architecture.
- Size-Variable Search Scheme: This "++" feature allows the algorithm to dynamically adjust cell depth and convolution kernel sizes. As shown in the architecture diagram below, smaller kernels inherit weights from the center of larger kernels, enabling the exploration of multiple model scales (Small, Medium, Large) in a single search pass without additional training costs.
Experimental Results
The researchers evaluated ZO-DARTS++ on the MedMNIST benchmark, comprising 10 diverse 2D medical imaging datasets such as PathMNIST and OrganAMNIST.
Efficiency and Accuracy
ZO-DARTS++ proved to be 38.6% faster than the original DARTS and outperformed established baselines like ResNet-18 and Google AutoML. The "Large" variant achieved an average accuracy of 84.00%, a notable increase over the 82.20% achieved by standard DARTS. Furthermore, the use of sparsemax allowed for early stopping at 40 epochs, further reducing computational overhead.
Model Compactness
A key strength of the algorithm is its ability to produce resource-efficient models. The "Small" variant reduced parameter counts by over 35% compared to standard NAS models while maintaining competitive performance. The distribution of these parameters across the various MedMNIST datasets highlights the algorithm's ability to adapt model complexity to the specific task at hand.
Conclusion
ZO-DARTS++ represents a significant step forward in making NAS practical for real-world applications. By combining gradient estimation efficiency with flexible model sizing, it provides a robust tool for discovering high-performance architectures for medical edge devices. Future work will focus on extending the framework to 3D medical data (e.g., CT/MRI scans) and automating the tuning of size-constraint hyperparameters.
Summary updated