Abstract
Neural Architecture Search (NAS) typically focuses on maximizing model accuracy, largely ignoring the massive energy and environmental costs of training thousands of candidate models. This paper introduces CNNGen, a grammar-based tool that generates diverse CNN architectures from scratch, alongside a new dataset of 1,300 models annotated with both performance and carbon emission metrics. Furthermore, the authors propose three lightweight predictors that can estimate a model's accuracy and energy consumption without the need for training, paving the way for greener AI development.
Introduction
Finding the optimal neural network architecture for a specific task—a process known as Neural Architecture Search (NAS)—is currently treated as a massive optimization problem. To solve it, NAS algorithms typically train and evaluate thousands of model variants. This process is incredibly resource-intensive and carries a significant carbon footprint.
To help researchers build better NAS algorithms without wasting compute, the community relies on benchmark datasets like NASBench-101, which provide pre-computed accuracies for hundreds of thousands of models. However, existing benchmarks suffer from two major flaws: they lack energy metrics, making it impossible to optimize for environmental impact, and they offer limited diversity, as they usually generate models by slightly mutating a single reference architecture.
The CNNGen Framework
To address these limitations, the authors developed CNNGen, a generator that uses a context-free grammar to build valid CNN topologies from scratch. Instead of tweaking a predefined cell, CNNGen's grammar defines the rules of how layers can be stacked, allowing it to produce highly diverse and novel architectures. CNNGen automatically compiles these designs into executable Keras code and uses CodeCarbon to track CO2 equivalents and total energy (kWh) during training.
The CNNGen Dataset
The authors produced a new benchmark dataset consisting of 1,300 randomly generated CNN models. For each model, the dataset provides the grammar phrase, Python code, a PNG visualization, accuracy, and energy metrics. These models were trained for 36 epochs on three standard image classification datasets: CIFAR-10, CIFAR-100, and Fashion-MNIST.
Zero-Cost Predictors
Because training models during a NAS search is exactly what researchers want to avoid, the authors developed three "zero-cost" predictors to estimate a model's accuracy and energy use based purely on its structure:
- Img Predictor: Analyzes a generated PNG diagram of the network's architecture.
- Py_code Predictor: Parses the generated Python source code to extract layer types and hyperparameters.
- Decision Tree (DT) Predictor: A simple regressor using four manually defined features: number of layers, number of epochs, FLOPs, and total parameter count.
Key Insights and Results
Diversity and Trade-offs
A core finding is that diversity is crucial for learning trade-offs. Because NASBench-101 uses cell-based generation, its models have a very tight accuracy distribution (mostly clustered between 80% and 85%). In contrast, CNNGen's grammar-based approach yields a much wider spread (40% to 68% on CIFAR-10). While the peak accuracy is lower, this wide dispersion is a prerequisite for training robust predictors that understand the full spectrum of design choices and complex energy-performance trade-offs.
Prediction Performance
- Accuracy Prediction: The
Py_codepredictor achieved the best results for forecasting model performance, showing the lowest Mean Absolute Error (MAE) and highest Kendall's $\tau$ correlation. This suggests that parsing actual Python code to extract hyperparameters is more effective than visual graph analysis for accuracy estimation. - Energy Prediction: The simple Decision Tree (
DT) predictor dominated the energy prediction task. It achieved significantly lower MAE and higher Kendall's $\tau$ across all three datasets, outperforming more complex code- or image-based predictors. This indicates that basic structural features (layers, FLOPs, parameters) are the primary drivers of energy consumption.
Conclusion
CNNGen provides a necessary toolset for the NAS community to move toward environmentally conscious model design. By providing a diverse dataset with energy annotations and demonstrating that energy consumption can be accurately predicted using simple structural features, this work enables the development of NAS algorithms that optimize for both accuracy and sustainability. Future work will involve integrating these zero-cost predictors directly into active NAS search algorithms to guide the discovery of greener neural networks.
Summary updated