Heed the noise in performance evaluations in neural architecture search

Arkadiy Dushatskiy, Tanja Alderliesten, Peter A. N. Bosman

Introduction

Neural Architecture Search (NAS) has revolutionized the design of deep learning models by automating the discovery of optimal architectures. However, a significant hurdle remains: stochastic noise. In many NAS implementations, a candidate architecture is evaluated based on a single training run. This performance metric is often unstable, influenced by random weight initialization (seeds), data splitting, and stochastic augmentations. This issue is particularly acute in Medical Image Segmentation (MIS), where small datasets amplify the impact of noise, leading search algorithms to select "lucky" architectures that do not generalize well.

Methodology: Robust Evaluation Setups

To address this, the researchers investigated the trade-off between the number of architectures explored and the reliability of their evaluation. They defined three distinct evaluation setups under a fixed total computational budget:

  1. Setup-1Fold: The standard baseline using a single training run on one data split.
  2. Setup-CV: 5-fold cross-validation, averaging five training runs.
  3. Setup-3CV: 5-fold cross-validation repeated three times with different splits, averaging 15 training runs.
The three evaluation setups used to compare reliability against computational cost.

The search space was centered on the U-Net architecture, allowing for variations in both topology and cell types (e.g., VGG, ResNet, Xception, EfficientNet blocks). Four search algorithms—Local Search, GOMEA, TPE, and SAGOMEA—were evaluated across two benchmarks: the Prostate Segmentation dataset (30 scans) and the ACDC dataset (100 scans).

The Impact of Noise: The Top 20% Trap

A critical finding of the study is the lack of rank correlation among high-performing models. While overall performance might appear correlated across different seeds, this relationship breaks down for the best-performing architectures.

Spearman rank correlation between different seeds and dataset splits. Note the near-zero correlation in the zoomed-in top-performing regions.

For the top 20% of architectures, the Spearman rank correlation between different random seeds was found to be nearly zero (rho < 0.1). This implies that a model ranked first using one seed might rank poorly with another, making single-run evaluations highly unreliable for identifying truly superior architectures.

Results: Quality over Quantity

The experiments demonstrated that investing in more reliable evaluations (Setup-CV) consistently yielded better final architectures than evaluating more candidates with a single fold (Setup-1Fold), even when the total training budget remained identical.

  • Search-Space Overfitting: Setup-1Fold often led to overfitting the search process to the specific noise of a single data split.
  • Performance Gains: Moving to Setup-CV significantly improved architecture quality across almost all algorithms. The best NAS-discovered models outperformed established baselines, including nnUnet on the Prostate dataset (0.726 vs. 0.699 Dice score).
  • Surrogate Efficiency: Surrogate-assisted algorithms like SAGOMEA proved particularly adept at handling the higher cost of reliable evaluations by effectively modeling the performance landscape.

Conclusion

This research highlights that in NAS, more data (in the form of repeated evaluations) is often better than more search iterations. By prioritizing evaluation reliability through cross-validation, researchers can mitigate the effects of stochastic noise and discover architectures that are genuinely robust. Future work will look toward extending these findings to 3D volumetric segmentation and exploring advanced cost-reduction techniques like learning curve modeling.

Summary updated

BibTeX
@inproceedings{dushatskiy2022heed,
  author = {Arkadiy Dushatskiy and Tanja Alderliesten and Peter A. N. Bosman},
  title = {{Heed the noise in performance evaluations in neural architecture search}},
  booktitle = {GECCO Companion},
  year = {2022},
  doi = {10.1145/3520304.3533995},
}
Figure