EnvPipe: Performance-preserving DNN Training Framework for Saving Energy

Sangjin Choi, Inhoe Koo, Jeongseob Ahn, Myeongjae Jeon, Youngjin KwonUSENIX ATC 2023 · OpenReview

TL;DR

This paper introduces ENVPIPE, a novel framework for energy-efficient DNN training that significantly reduces energy consumption without sacrificing accuracy or performance. By intelligently exploiting "pipeline bubbles" in synchronous pipeline parallelism and dynamically adjusting GPU core frequencies, ENVPIPE achieves substantial energy savings, making large-scale DNN training more sustainable.

What Problem Does This Paper Address?

The ever-increasing demand for deep neural networks (DNNs) has led to massive computational workloads, particularly in training large models. These workloads are a significant contributor to the energy consumption of data centers, raising concerns about their environmental impact and operational costs. While techniques like GPU Dynamic Voltage and Frequency Scaling (DVFS) exist to save energy, they often come at the cost of reduced performance or require complex hyperparameter tuning, which can be impractical for many users. Furthermore, existing methods often focus on single-GPU training and don't adequately address the challenges of large-scale, multi-GPU distributed training, where energy consumption is amplified. The core challenge is to achieve significant energy savings in these demanding training scenarios without compromising the critical aspects of accuracy and training speed.

Methods

ENVPIPE tackles this challenge by introducing a performance-preserving energy-saving framework for multi-GPU DNN training using synchronous pipeline parallelism (S-PP). Its key innovations include:

  • Exploiting Pipeline Bubbles: Unlike previous work that views "pipeline bubbles" (idle time in S-PP) as an inefficiency to be minimized, ENVPIPE treats them as an opportunity for energy savings. It categorizes bubbles into "usable" and "unusable" based on whether stretching a pipeline unit would delay subsequent units.
  • Selective SM Frequency Scaling: ENVPIPE dynamically adjusts the Streaming Multiprocessor (SM) frequency of GPUs. It identifies non-critical pipeline units and lowers their SM frequency to the optimal point identified by an "energy valley curve" (the frequency that offers the best energy efficiency). This is done without altering hyperparameters, thus preserving accuracy.
  • Performance-Critical Path Identification: To avoid performance degradation, ENVPIPE identifies the performance-critical path of the pipeline. It ensures that SM frequency is not reduced on this path, or if it is, it reconfigures frequencies in small, iterative steps to maintain performance.
  • Online Profiling and Scheduling: The framework includes an online profiler to determine the energy valley curve for each pipeline stage and a scheduler that reschedules pipeline units to maximize the creation and utilization of usable bubbles.

Datasets

The paper does not explicitly mention specific datasets used for training the DNN models. Instead, it focuses on evaluating ENVPIPE's performance on various established DNN models, including:

  • Transformer-based models: BERT (BERT-336M, BERT-1.3B, BERT-3.9B), GPT (GPT-125M, GPT-350M), Megatron (Megatron-125M, Megatron-350M, Megatron-760M).
  • CNN models: ResNet (ResNet-152).

These models are used in both single-node (4 GPUs) and multi-node (16 GPUs) training configurations.

Key Insights and Ideas

  • Bubbles as Opportunities: The most significant insight is reframing pipeline bubbles not as a problem to be solved, but as a resource to be exploited for energy savings. This paradigm shift is central to ENVPIPE's approach.
  • Usable vs. Unusable Bubbles: The distinction between usable and unusable bubbles is crucial. Only by targeting usable bubbles can energy savings be achieved without performance degradation.
  • Side-Effect-Free Control: ENVPIPE's reliance on SM frequency as the sole control knob is key to preserving accuracy and avoiding the complexities of hyperparameter tuning. This makes it broadly applicable.
  • Iterative Reconfiguration: The iterative approach to identifying and reconfiguring the critical path is essential for adapting to dynamic changes and ensuring that performance is not compromised.

Key Results

ENVPIPE demonstrates significant energy savings with minimal performance impact:

  • Single-Node Training:
    • Up to 25.2% energy saving on a single-node setup with 4 GPUs.
  • Performance degradation is less than 1%.
  • Multi-Node Training:
    • Up to 28.4% energy saving on a multi-node setup with 16 GPUs.
  • Performance degradation remains less than 1%.
  • Comparison to Baselines: ENVPIPE consistently outperforms baseline methods (e.g., Uniform, NoRecfg) in terms of energy savings while maintaining performance.

Limitations and Future Work

  • Microbatch Size Impact: The paper notes that the effectiveness of energy savings can be influenced by the number of microbatches. Larger numbers of microbatches can lead to more effective energy savings, especially for models with larger pipeline bubbles, but this is limited by global batch size constraints for maintaining statistical efficiency.
  • Stage Partitioning: Different stage partitioning methods can affect energy savings. Balancing execution time generally leads to higher throughput, but the energy savings are similar to balancing memory footprint, with the former showing slightly better results.
  • Future Work: The authors suggest that ENVPIPE's approach can be extended to other parallelism techniques beyond pipeline parallelism, and further research into optimizing stage partitioning for various model architectures could yield even greater energy savings. The framework's modular design also allows for easier adaptation to new ML platforms.

Summary updated

BibTeX
@inproceedings{choi2023envpipe,
  author = {Sangjin Choi and Inhoe Koo and Jeongseob Ahn and Myeongjae Jeon and Youngjin Kwon},
  title = {{EnvPipe: Performance-preserving DNN Training Framework for Saving Energy}},
  booktitle = {USENIX ATC},
  year = {2023},
  url = {https://openreview.net/forum?id=SCzyDN1zuJ},
}
Figure