Precise Energy Consumption Measurements of Heterogeneous Artificial Intelligence Workloads

René Caspart, Sebastian Ziegler, Arvid Weyrauch, Holger Obermaier, Simon Raffeiner, Leon Pascal Schuhmacher, Jan M. Scholtyssek, Darya Trofimova, Marco Nolden, Ines Reinartz, Fabian Isensee, Markus Götz, Charlotte DebusLecture Notes in Computer Science 2022 · DOI · OpenAlex

Abstract

With the rise of AI in recent years and the increase in complexity of the models, the growing demand in computational resources is starting to pose a significant challenge. The need for higher compute power is being met with increasingly more potent accelerators and the use of large compute clusters. However, the gain in prediction accuracy from large models trained on distributed and accelerated systems comes at the price of a substantial increase in energy demand, and researchers have started questioning the environmental friendliness of such AI methods at scale. Consequently, energy efficiency plays an important role for AI model developers and infrastructure operators alike. The energy consumption of AI workloads depends on the model implementation and the utilized hardware. Therefore, accurate measurements of the power draw of AI workflows on different types of compute nodes is key to algorithmic improvements and the design of future compute clusters and hardware. To this end, we present measurements of the energy consumption of two typical applications of deep learning models on different types of compute nodes. Our results indicate that 1. deriving energy consumption directly from runtime is not accurate, but the consumption of the compute node needs to be considered regarding its composition;

  1. neglecting accelerator hardware on mixed nodes results in overproportional inefficiency regarding energy consumption;
  2. energy consumption of model training and inference should be considered separately - while training on GPUs outperforms all other node types regarding both runtime and energy consumption, inference on CPU nodes can be comparably efficient. One advantage of our approach is that the information on energy consumption is available to all users of the supercomputer, enabling an easy transfer to other workloads alongside a raise in user-awareness of energy consumption.

Introduction

As artificial intelligence models grow in complexity, their environmental impact has become a focal point for the "Green AI" movement. Traditionally, researchers estimate energy consumption using rough proxies, such as multiplying a model's runtime by the hardware's Thermal Design Power (TDP). However, this approach fails to account for the energy consumed by the entire node (including CPU, RAM, and cooling) and the specific power profiles of heterogeneous hardware. This paper addresses these gaps by providing precise, whole-node energy measurements on a shared supercomputer without requiring administrative privileges.

Measurement Methodology

The study was conducted on the HoreKa supercomputer, utilizing two distinct node types: HoreKa-Green (equipped with four NVIDIA A100 GPUs) and HoreKa-Blue (CPU-only). To overcome the "root access barrier," the researchers leveraged the XClarity Controller (XCC) to access internal hardware sensors. These sensors were integrated into the Slurm workload manager via a custom plugin, allowing standard users to view the exact energy cost of their jobs in the accounting database upon completion. The study compared four operational scenarios: exclusive GPU use, CPU-mix (running on CPUs of a GPU-equipped node), standard CPU-only nodes, and interactive Jupyter notebook usage.

Use Cases and Model Complexity

Two distinct AI workloads were evaluated to represent common scientific tasks:

  1. Use Case Energy: A time-series regression task using an LSTM model to forecast electricity demand based on the Western Europe Power Consumption Dataset.
  2. Use Case Health: An image classification task using a VGG-19 architecture to detect COVID-19 infections from lung X-ray images.
Table 1: Model parameters and computational complexity (FLOPs) for the evaluated use cases.

The models vary significantly in scale, with the CNN used in the Health use case being orders of magnitude more complex than the LSTM used for Energy forecasting, as shown in the architectural complexity of the COVID-19 detection model.

Figure 1: CNN architecture for the Health use case, performing binary classification of COVID-19 infection from X-ray images.

Key Findings

The Inefficiency of Mixed Nodes

A critical finding is the "Idle GPU" penalty. Running CPU-only workloads on nodes equipped with GPUs is highly inefficient because idle GPUs still draw significant baseline power. For the LSTM task, the "CPU-mix" setup consumed 1.7x more energy than a dedicated "CPU-only" node for the exact same workload.

Runtime as an Inaccurate Proxy

The relationship between runtime and energy consumption is non-linear. In the Health use case, training on a CPU took 194x longer than on a GPU but consumed only 86x more energy. This discrepancy highlights that energy consumption must be measured directly rather than inferred from time, as hardware-specific power profiles vary significantly.

Training vs. Inference Efficiency

While GPUs are the undisputed leaders in training efficiency—being 7.4x faster and using 75% less energy for the LSTM task—the gap narrows during inference. For LSTM inference, CPU-only nodes were found to be "comparably efficient" (189.5 kJ vs 156.9 kJ for GPU), suggesting that CPUs are a viable deployment option when latency is not the primary constraint.

Conclusion

This research demonstrates that providing transparent energy metrics to supercomputer users is both feasible and necessary. By accounting for whole-node consumption and hardware heterogeneity, developers can make informed decisions about where to run their workloads. Future work will focus on mapping energy consumption to specific model components, such as self-attention layers, to further refine energy-aware AI development.

Summary updated

BibTeX
@inproceedings{caspart2022precise,
  author = {René Caspart and Sebastian Ziegler and Arvid Weyrauch and Holger Obermaier and Simon Raffeiner and Leon Pascal Schuhmacher and Jan M. Scholtyssek and Darya Trofimova and Marco Nolden and Ines Reinartz and Fabian Isensee and Markus Götz and Charlotte Debus},
  title = {{Precise Energy Consumption Measurements of Heterogeneous Artificial Intelligence Workloads}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2022},
  doi = {10.1007/978-3-031-23220-6\_8},
}

Loading related content...

Figure