GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks

Gabriel Cortês, Nuno Lourenço, Paolo Romano, Penousal MachadoarXiv 2025 · arXiv

TL;DR

GreenFactory is a novel ensemble method that combines multiple "zero-cost proxies" using a random forest regressor to directly predict a neural network's final test accuracy without any training. By leveraging the complementary strengths of 21 different training-free metrics, it significantly outperforms individual proxies, achieving exceptional generalization and state-of-the-art ranking correlations across diverse network architectures and datasets.

What Problem Does This Paper Address?

Neural Architecture Search (NAS) has revolutionized deep learning by automating the design of neural networks, but it comes with a massive environmental and computational price tag. Fully training and evaluating thousands of candidate networks consumes immense amounts of GPU time and electricity.

To mitigate this, researchers developed zero-cost proxies—mathematical formulas or algorithms that estimate a network's performance at initialization (before any training occurs) using just a single mini-batch of data. However, existing zero-cost proxies suffer from two major flaws:

  1. Poor Generalization: A proxy that performs exceptionally well on one dataset or search space often fails completely on another.
  2. Relative Rankings Only: They generate abstract scores that can rank models relative to one another but cannot predict the actual expected accuracy of a network.

The authors introduce GreenFactory to bridge this gap by ensembling these isolated proxies into a unified, robust predictor.

Methods

GreenFactory treats performance estimation as a supervised regression problem.

  1. Feature Collection: For a given network architecture, the authors compute 21 different zero-cost proxies from the literature (including NASWOT, Synflow, GradNorm, TE-NAS, Zen-NAS, ZiCo, EZ-NAS, AZ-NAS variants, and 10 evolutionarily discovered GreenMachine proxies). They complement these with basic network metadata: the number of trainable parameters and FLOPs.
  2. Model Selection: After evaluating multiple regression algorithms (including Linear Regression, Decision Trees, and Gradient Boosting), a Random Forest Regressor combined with a stratified sampling strategy was selected because it excelled at capturing complex, non-linear interactions between the proxy scores.
  3. Feature Elimination (RFE): To prevent the computational overhead of calculating 21 distinct proxies, the authors used Recursive Feature Elimination to find optimal trade-offs between prediction error (RMSE) and compute time.

This process yielded two distinct configurations:

  • GreenFactory (GF): Uses 25 features to achieve the absolute lowest error.
  • GreenFactory-Fast (GF-Fast): Uses just 6 key features, slashing computational time by roughly 90% while maintaining highly competitive accuracy.

Datasets

The approach was trained and evaluated using NATS-Bench, a well-established NAS benchmark consisting of two distinct search spaces:

  • NATS-Bench-SSS (Size Search Space): 32,768 candidate architectures varying in channel sizes.
  • NATS-Bench-TSS (Topology Search Space): 15,625 candidate architectures varying in cell connections.

Each architecture in the benchmark contains ground-truth performance data across three major image classification datasets: CIFAR-10, CIFAR-100, and ImageNet-16-120.

Histograms of test accuracy across NATS-Bench search spaces and datasets.

Key Insights and Ideas

  • The Power of Non-Linear Ensembling: Linear models fail to combine zero-cost proxies effectively because the relationships between proxy scores and actual accuracy are highly non-linear. The Random Forest model succeeds because it can isolate specific conditions where certain proxies excel.
  • Evolutionary Proxies Carry the Weight: When analyzing feature importance, proxies discovered via evolutionary algorithms (like GreenMachine and EZ-NAS) dominated the model's decision-making process. In the full GreenFactory model, these proxies accounted for a massive portion of the overall feature importance.
Feature importance for the optimized GreenFactory and GreenFactory-Fast models.
  • Diminishing Returns in Feature Selection: As shown during the feature elimination process, adding more than 6 features yields minor improvements in Root Mean Squared Error (RMSE) but causes an exponential spike in the time required to compute the proxies. This justifies the creation of the "Fast" variant for real-world pipelines.
Relationship between the number of selected features, RMSE, and computation time.

Key Results

GreenFactory directly predicts absolute test accuracy and achieves state-of-the-art rank consistency, measured via Kendall’s ($\tau$) and Spearman’s ($\rho$) correlation coefficients.

  • NATS-Bench-SSS (Size Search Space): GreenFactory achieved exceptional Kendall correlations of 0.907 for CIFAR-10, 0.945 for CIFAR-100, and 0.920 for ImageNet-16-120.
  • NATS-Bench-TSS (Topology Search Space): It maintained high reliability with Kendall correlations of 0.921 for CIFAR-10, 0.929 for CIFAR-100, and 0.908 for ImageNet-16-120.
  • Consistency: GreenFactory consistently outperformed every single individual zero-cost proxy across all datasets and search spaces, proving its superior generalization capabilities.

Limitations and Future Work

While highly effective, GreenFactory's primary limitation is its reliance on NATS-Bench for training data, which restricts its current evaluation to specific image classification tasks and architectures. Additionally, the model utilizes one-hot encoding to identify the target dataset, meaning it cannot automatically generalize to a completely unseen dataset without retraining the regressor.

Future Directions: The authors propose expanding evaluation to more diverse benchmarks beyond NATS-Bench. Crucially, they suggest replacing one-hot dataset indicators with abstract meta-features (e.g., input/output dimensions, number of classes, task type) so that GreenFactory can predict network performance on entirely new datasets out-of-the-box.

Summary updated

Abstract

Determining the performance of a Deep Neural Network during Neural Architecture Search processes is essential for identifying optimal architectures and hyperparameters. Traditionally, this process requires training and evaluation of each network, which is time-consuming and resource-intensive. Zero-cost proxies estimate performance without training, serving as an alternative to traditional training. However, recent proxies often lack generalization across diverse scenarios and provide only relative rankings rather than predicted accuracies. To address these limitations, we propose GreenFactory, an ensemble of zero-cost proxies that leverages a random forest regressor to combine multiple predictors' strengths and directly predict model test accuracy. We evaluate GreenFactory on NATS-Bench, achieving robust results across multiple datasets. Specifically, GreenFactory achieves high Kendall correlations on NATS-Bench-SSS, indicating substantial agreement between its predicted scores and actual performance: 0.907 for CIFAR-10, 0.945 for CIFAR-100, and 0.920 for ImageNet-16-120. Similarly, on NATS-Bench-TSS, we achieve correlations of 0.921 for CIFAR-10, 0.929 for CIFAR-100, and 0.908 for ImageNet-16-120, showcasing its reliability in both search spaces.
BibTeX
@misc{cortes2025greenfactory,
  author = {Gabriel Cortês and Nuno Lourenço and Paolo Romano and Penousal Machado},
  title = {{GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks}},
  year = {2025},
  eprint = {2505.09344},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.09344},
}
Figure