Abstract
As deep learning models grow in complexity, their environmental footprint has become a critical concern. This paper introduces a "Green AutoML" framework designed to optimize Deep Shift Neural Networks (DSNNs) for both predictive performance and energy efficiency. By employing multi-objective hyperparameter optimization (HPO) via the SMAC3 framework and integrating real-time carbon emission tracking, we demonstrate that DSNNs can achieve high accuracy (over 84%) while minimizing $gCO_2eq$ emissions. Our approach utilizes multi-fidelity techniques, specifically identifying "Shift Depth" as a proxy for model fidelity, to significantly reduce the computational cost of the search process.
Introduction
The massive computational requirements of modern Deep Learning (DL) lead to substantial energy consumption and carbon emissions. Deep Shift Neural Networks (DSNNs) offer a promising alternative by replacing power-hungry floating-point multiplications with hardware-efficient bit-shift operations. However, DSNNs are notoriously difficult to tune; selecting the optimal bit-width, precision, and number of shift layers involves navigating a complex trade-off between accuracy and computational load. Manually searching this high-dimensional space is inefficient and often fails to find the most sustainable configurations.
This research addresses the need for an automated, multi-objective approach to navigate the "accuracy vs. sustainability" trade-off in DSNN design, ensuring that environmental impact is a primary consideration rather than an afterthought.
Methodology
The proposed framework leverages the SMAC3 (Sequential Model-based Algorithm Configuration) framework to automate the optimization of DSNNs. The methodology is built on two core pillars:
1. Multi-Objective (MO) Optimization
Rather than optimizing solely for accuracy, the algorithm treats Top-1 accuracy and energy consumption (measured in $gCO_2eq$) as competing objectives. We utilize the ParEGO algorithm to identify a "Pareto front"—a set of configurations where no single objective can be improved without degrading the other. This allows practitioners to choose models based on their specific constraints for performance and sustainability.
2. Multi-Fidelity (MF) Optimization
To accelerate the search, we implement HyperBand and introduce a novel fidelity metric: Shift Depth. Training a full DSNN is time-consuming; however, we hypothesized that models performing well with fewer shift layers (lower numerical uncertainty) would likely remain top performers in the full architecture. By starting with lower shift depths and only promoting the most promising candidates to full training, we drastically reduce the total search budget.
To ensure accurate environmental monitoring, we integrated CodeCarbon into the training loop. This tool tracks real-time energy usage and accounts for regional grid carbon intensity, providing a transparent metric for sustainability.
Experimental Setup
The experiments were conducted using the CIFAR-10 dataset, a standard benchmark for image classification. We utilized a ResNet20 architecture as the base, transforming its standard layers into DSNN-compatible shift layers. The search space included parameters such as batch size, optimizer type (e.g., SGD, Adam, Ranger), learning rate, momentum, and DSNN-specific parameters like weight bits and activation precision.
Key Results and Discussion
Our findings demonstrate that automated optimization significantly outperforms default configurations in both performance and efficiency.
Accuracy and Pareto Efficiency
The optimized DSNN configurations achieved a Top-1 accuracy of 84.67% on CIFAR-10, representing a boost of over 3% compared to unoptimized "out-of-the-box" settings. The search identified two distinct Pareto-optimal solutions (POS):
- POS 1: Prioritizes efficiency, achieving 83.50% accuracy with 0.1661 $gCO_2eq$ emissions.
- POS 2: Prioritizes performance, reaching 84.67% accuracy with a negligible increase in emissions to 0.1673 $gCO_2eq$.
Search Efficiency
The multi-fidelity approach proved highly effective. The algorithm was able to identify these optimal configurations after evaluating only 33 to 50 unique configurations. This efficiency is crucial for "Green AI," as the energy saved during the search process itself contributes to the overall sustainability of the model development lifecycle.
Conclusion and Future Work
This study confirms that "Green AutoML" is not only feasible but essential for the future of sustainable deep learning. By treating carbon emissions as a first-class citizen in the HPO process, we can develop models that are both powerful and environmentally responsible.
Future research will focus on scaling these techniques to larger datasets like ImageNet and more complex architectures such as Transformers. Additionally, we aim to refine the integration of ParEGO and HyperBand to further reduce the computational overhead of calculating Pareto fronts and to incorporate hardware-specific energy profiles into the optimization loop.
Summary updated