An Evaluation of Zero-Cost Proxies - From Neural Architecture Performance Prediction to Model Robustness

Jovita Lukasik, Michael Moeller, Margret KeuperDAGM 2023 · DOI

Abstract

Neural Architecture Search (NAS) has significantly advanced the automation of high-performance network design, yet the computational cost of evaluating candidate architectures remains a primary bottleneck. Zero-Cost Proxies (ZCPs) have emerged as a solution, estimating performance at initialization without training. While ZCPs are effective for predicting standard "clean" accuracy, their utility in forecasting adversarial robustness is less understood. This paper evaluates the predictive power of 15 ZCPs across the NAS-Bench-201 search space. Our findings reveal that while clean accuracy can be predicted with high precision ($R^2$ up to 0.97) using a few key proxies, predicting robustness is a more complex task that requires an ensemble of metrics, achieving lower but still significant predictive performance ($R^2 \approx 0.65$).

Introduction

The "classical" approach to Neural Architecture Search (NAS) is notoriously resource-intensive, often requiring thousands of GPU hours to train and evaluate thousands of candidate architectures. To mitigate this, researchers introduced Zero-Cost Proxies (ZCPs)—mathematical metrics that analyze a network's weights and gradients at initialization to estimate its eventual performance.

While ZCPs have proven successful in identifying architectures with high standard accuracy, modern deep learning demands models that are also robust against adversarial attacks—subtle, intentional perturbations designed to cause misclassification. This study addresses a critical gap in the literature: Can these "free" metrics also serve as reliable indicators of a model's robustness, or are they limited to predicting standard performance?

Methodology

Rather than proposing a new proxy, this research conducts a comprehensive meta-evaluation of existing metrics. The authors utilize a Random Forest regressor to model the relationship between ZCPs and model performance.

Feature Collection and Prediction

The study gathered 15 distinct ZCPs, categorized into three primary groups:

  1. Jacobian-based: Metrics that measure the sensitivity of the output with respect to the input (e.g., jacov, nwot).
  2. Pruning-based: Metrics originally designed to identify important parameters for network pruning (e.g., snip, grasp, synflow).
  3. Architectural Baselines: Simple metrics such as FLOPs and parameter counts.

These 15 proxies served as input features for a Random Forest model. The model was tasked with predicting two distinct targets: "Clean" test accuracy and "Robust" accuracy under various adversarial conditions.

Evaluation Metrics

To understand the contribution of each proxy, the researchers employed Gini Importance and Permutation Importance. This allowed for a granular analysis of which architectural features are most indicative of robustness versus standard accuracy.

Experimental Setup

The evaluation was conducted using three foundational NAS benchmarking resources:

  • NAS-Bench-201: A standardized search space containing 6,466 unique architectures.
  • NAS-Bench-Suite-Zero: A dataset providing pre-computed ZCP values for the architectures in NAS-Bench-201.
  • Robustness Dataset (Jung et al., 2023): A comprehensive evaluation of NAS-Bench-201 architectures against four adversarial attacks: FGSM, PGD, APGD, and Square Attack. These were tested across CIFAR-10, CIFAR-100, and ImageNet16-120.

Key Insights

The Complexity of Robustness

A primary finding of this study is that robustness is a "team effort." While clean accuracy can often be predicted with high accuracy using only one or two dominant proxies (such as jacov or nwot), predicting robustness requires the collective information of nearly all 15 proxies. This suggests that adversarial robustness is a multi-faceted trait that cannot be captured by a single initialization-time metric.

The Dominance of Jacobian Proxies

Proxies based on the Jacobian matrix—which captures first-order partial derivatives—consistently provided the most useful information for both clean and robust performance. However, the specific importance of these features shifts depending on the target. For example, while jacov is the primary predictor for clean accuracy on CIFAR-10, the pruning-based proxy snip becomes significantly more important when predicting robustness against FGSM attacks.

The Transferability Gap

The study highlights a notable gap between accuracy and robustness. While ZCPs can predict both, the correlation for robustness is consistently weaker. The Random Forest achieved an $R^2$ score of 0.93 to 0.97 for clean accuracy, but this dropped to approximately 0.65 for PGD attacks on CIFAR-10. This indicates that architectures optimized solely for clean accuracy via ZCPs may not inherently possess adversarial resilience.

Results and Discussion

The meta-evaluation demonstrates that ZCPs are viable for multi-objective NAS. When the Random Forest was trained to predict both clean and robust accuracy simultaneously, it achieved $R^2$ scores between 0.81 and 0.86 for various attacks on CIFAR-10.

Interestingly, the study identified the zen proxy as an outlier. Despite its popularity in recent NAS literature, it showed the lowest correlation with test accuracy within the NAS-Bench-201 search space, suggesting that the effectiveness of certain proxies may be highly dependent on the specific search space structure.

Conclusion and Future Work

This research confirms that Zero-Cost Proxies contain valuable signals for predicting model robustness, though the relationship is more complex than it is for standard accuracy. The necessity of using an ensemble of proxies for robustness prediction suggests that future NAS frameworks should move away from single-proxy evaluations.

Limitations of this study include its focus on the cell-based NAS-Bench-201 search space. Future research should investigate whether these findings generalize to "macro" search spaces with more varied global structures. Additionally, there is a clear opportunity to develop new ZCPs specifically engineered to capture adversarial vulnerability at initialization, rather than relying on proxies designed for standard performance estimation.

Summary updated

BibTeX
@inproceedings{lukasik2023evaluation,
  author = {Jovita Lukasik and Michael Moeller and Margret Keuper},
  title = {{An Evaluation of Zero-Cost Proxies - From Neural Architecture Performance Prediction to Model Robustness}},
  booktitle = {DAGM},
  year = {2023},
  doi = {10.1007/978-3-031-54605-1\_40},
}
Figure