Introduction
Neural Architecture Search (NAS) has revolutionized the design of deep learning models by automating the discovery of optimal network topologies. Among NAS methodologies, Differentiable Architecture Search (DARTS) has emerged as a dominant paradigm due to its efficiency. By relaxing the discrete search space into a continuous one, DARTS allows for optimization via gradient descent, significantly reducing the computational burden compared to reinforcement learning or evolutionary algorithms.
However, DARTS is plagued by a significant "discretization gap." During the search phase, the algorithm optimizes a "supernet" containing a weighted combination of all possible operations. Once the search concludes, the supernet is discretized by selecting only the highest-weighted operations. Recent studies have identified that the supernet often converges to sharp minima—regions of the loss landscape where minor perturbations in parameters (such as those induced by discretization) lead to catastrophic drops in performance. While some methods attempt to smooth this landscape using adversarial perturbations, they often introduce substantial computational overhead.
Self-Distillation DARTS (SD-DARTS)
To address the discretization gap without the cost of complex perturbations, we introduce Self-Distillation DARTS (SD-DARTS). This approach leverages the supernet’s own training history to regularize the search process, effectively flattening the loss landscape.
Self-Distillation Regularization
The core mechanism of SD-DARTS is a distillation loss applied during the search phase. At any given epoch $t$, the supernet from the previous epoch $t-1$ serves as a "teacher." The current supernet (the "student") is penalized via Kullback-Leibler (KL) divergence if its output probability distributions deviate from those of the teacher. This temporal consistency forces the optimization process to favor regions of the parameter space that are stable over time.
Mathematically, the authors demonstrate that minimizing the divergence between consecutive iterations is equivalent to minimizing the $l_2$-norm of the gradient. This relationship is critical, as a smaller gradient norm is a direct indicator of a flatter loss landscape, which is more robust to the discretization process.
Voting Teachers
To mitigate the noise inherent in a single previous epoch, SD-DARTS employs a "Voting Teacher" strategy. Instead of relying solely on epoch $t-1$, the algorithm maintains a sliding window of size $K$ containing the outputs from the most recent epochs. The teacher signal is then defined as the average of these predictions. Empirical results suggest that a window of $K=2$ provides the optimal balance; while an ensemble improves stability, using a window that is too large ($K > 5$) introduces stale information that can hinder the convergence of the current supernet.
Experimental Results
Benchmarks and Performance
SD-DARTS was evaluated across three major benchmarks to test its robustness and transferability:
- NAS-Bench-201: On this standardized benchmark, SD-DARTS achieved new state-of-the-art (SOTA) results across CIFAR-10, CIFAR-100, and ImageNet-16-120, proving its consistency across diverse datasets.
- CIFAR-10: The method achieved a mean test error of 2.58%, with a peak performance of 2.44%, surpassing standard DARTS (2.76%) and several more computationally intensive variants.
- ImageNet: Architectures discovered on CIFAR-10 transferred effectively to ImageNet, reaching a competitive top-1 error rate of 25.0%.
Landscape Analysis
To verify the theoretical claim of landscape flattening, the researchers calculated the dominant eigenvalue of the Hessian norm of the loss. SD-DARTS consistently yielded lower Hessian eigenvalues compared to standard DARTS. This empirical evidence confirms that self-distillation successfully guides the search toward flatter minima, directly narrowing the discretization gap.
Efficiency
One of the primary advantages of SD-DARTS is its minimal computational footprint. By simply storing logits from previous iterations, the method adds negligible memory and time overhead. The total search cost was recorded at approximately 0.37 GPU-days, maintaining the high-speed appeal of gradient-based NAS.
Conclusion and Future Work
SD-DARTS provides a simple yet highly effective solution to the discretization gap in differentiable architecture search. By utilizing self-distillation to enforce temporal consistency, the method ensures the supernet converges to flat minima, resulting in discrete architectures that retain high performance.
Future research may explore the use of Decayed Exponential Moving Averages (EMA) for the voting teacher signal. Rather than a simple average of the last $K$ steps, an EMA could provide a smoother, more sophisticated teacher signal that further enhances the stability of the search process.
Summary updated