SustainDC - Benchmarking for Sustainable Data Center Control

Avisek Naug, Antonio Guillen, Ricardo Luna, Vineet Gundecha, Desik Rengarajan, Sahand Ghorbanpour, Sajad Mousavi, Ashwin Ramesh Babu, Dejan Markovikj, Lekhapriya Dheeraj Kashyap, Soumyendu Sarkar

TL;DR

This paper introduces SustainDC, a novel Python-based benchmarking environment for developing and evaluating multi-agent reinforcement learning (MARL) algorithms for sustainable data center control. SustainDC allows for the simulation of complex data center operations, including workload scheduling, cooling optimization, and battery management, enabling researchers to benchmark MARL approaches for reducing energy consumption and carbon footprint.

What Problem Does This Paper Address?

The ever-increasing demand for computational power, driven by AI and cloud services, has led to a massive growth in data centers. These facilities are significant consumers of energy and contribute substantially to carbon emissions. While reinforcement learning (RL) has shown promise in optimizing complex systems, there's a lack of standardized, flexible, and realistic environments to benchmark MARL algorithms specifically for data center sustainability. Existing tools often lack customization, are difficult to integrate with ML frameworks, or don't fully capture the interconnectedness of data center operations. This paper aims to fill that gap by providing a comprehensive platform for developing and evaluating MARL strategies for greener data centers.

Methods

The core contribution is the SustainDC environment, built entirely in Python and compatible with the Gymnasium API. It models three key interconnected environments:

  • Workload Environment (EnvLS): Manages the scheduling of delayable workloads. RL agents can learn to defer tasks to periods of lower grid carbon intensity (CI) to reduce the overall carbon footprint.
  • Data Center Environment (EnvDC): Simulates the IT and HVAC systems. Agents learn to optimize cooling setpoints (e.g., CRAC supply air temperature) to balance IT equipment performance and energy consumption.
  • Battery Environment (EnvBAT): Models auxiliary battery systems. Agents learn to optimally charge and discharge batteries to leverage periods of low CI and provide auxiliary power during high CI periods.

SustainDC supports:

  • Customizable DC configurations: Users can define various aspects of the data center, including server specs, HVAC parameters, and physical layout, via a dc_config.json file.
  • Heterogeneous MARL: It accommodates agents with different observation and action spaces, crucial for complex systems like data centers.
  • Reward shaping: Allows for flexible reward formulations to prioritize different sustainability metrics.

The paper evaluates various MARL algorithms, including single-agent RL, IPPO with different reward sharing mechanisms, and heterogeneous MARL approaches like HAPPO, HAA2C, HAD3QN, and HASAC.

Datasets

SustainDC utilizes:

  • Open-source workload traces: Data from Alibaba (v2017) and Google (v2019) data centers are included. Users can also provide their own custom workload traces.
  • Location-specific Carbon Intensity (CI) data: Files are extracted from EIA.gov for various US states (Arizona, California, Georgia, Illinois, New York, Texas, Virginia, Washington).
  • Location-specific weather data: .epw files from EnergyPlus are used for the same locations.

The paper introduces SustainDC itself as a new benchmark environment for MARL in data center control.

Key Insights and Ideas

  • Interconnectedness is Key: The paper strongly emphasizes that optimizing individual components (workload, cooling, battery) in isolation is suboptimal. Coordinated MARL approaches are necessary to achieve significant carbon footprint reductions.
  • MARL Outperforms Single-Agent: Benchmarking shows that multi-agent systems, especially those with shared critics (like MAPPO and HAPPO), generally outperform single-agent RL or combinations of single-agent RL with baseline controllers.
  • Reward Sharing Matters: Ablation studies on the collaborative reward coefficient (α) indicate that sharing rewards among agents can improve performance, particularly in partially observable environments.
  • Algorithm Performance Varies by Location: No single MARL algorithm consistently performs best across all geographical locations and their unique weather and carbon intensity profiles. This highlights the need for robust and adaptable algorithms.
  • Customization is Crucial: The ability to customize DC configurations, workloads, and external variables (weather, CI) is vital for realistic benchmarking and for tailoring solutions to specific data center deployments.

Key Results

  • Significant CFP Reduction Potential: The paper demonstrates that MARL algorithms can significantly reduce the carbon footprint of data centers compared to baseline methods. For instance, in New York, HAPPO achieved a lower CFP than IPPO.
  • MARL vs. Baselines: In Table 2, the best MARL combination (ALS + ADC + ABAT) achieved a CFP of 155.23 kgCO2, outperforming several single-agent and baseline combinations.
  • HAPPO Shows Promise: Across various locations, HAPPO frequently demonstrated strong performance, often outperforming other MARL algorithms in terms of CFP and energy consumption. For example, in California (Table 6), HAPPO achieved a CFP of 159.1 kgCO2.
  • No One-Size-Fits-All: While HAPPO was often strong, HASAC showed superior performance in Arizona (Table 7), indicating the importance of location-specific tuning and algorithm selection.
  • Reward Sharing Benefits: IPPO with a collaborative reward coefficient of α=0.8 showed slightly better performance than α=1.0 (no collaboration) or α=0.1 (heavy reliance on other agents' rewards), suggesting a balanced approach is beneficial.

Limitations and Future Work

  • Oracle Absence: The paper acknowledges the difficulty in quantifying the absolute best performance due to the absence of an oracle that knows the optimal solution for all environments.
  • Hyperparameter Tuning: Exhaustively tuning hyperparameters for all networks was not feasible.
  • Future Work:
    • Deploying trained agents to real data centers with safeguards and domain adaptation.
  • Augmenting the codebase with updates for real-world deployment.
  • Investigating hierarchical RL algorithms.
  • Implementing continual reinforcement learning to handle out-of-distribution data center equipment changes.
  • Further exploring the variations in performance across different regions and their causes.
  • Collaborating on RL reward design for SustainDC.

Summary updated

Abstract

Machine learning has driven an exponential increase in computational demand, leading to massive data centers that consume significant amounts of energy and contribute to climate change. This makes sustainable data center control a priority. In this paper, we introduce SustainDC, a set of Python environments for benchmarking multi-agent reinforcement learning (MARL) algorithms for data centers (DC). SustainDC supports custom DC configurations and tasks such as workload scheduling, cooling optimization, and auxiliary battery management, with multiple agents managing these operations while accounting for the effects of each other. We evaluate various MARL algorithms on SustainDC, showing their performance across diverse DC designs, locations, weather conditions, grid carbon intensity, and workload requirements. Our results highlight significant opportunities for improvement of data center operations using MARL algorithms. Given the increasing use of DC due to AI, SustainDC provides a crucial platform for the development and benchmarking of advanced algorithms essential for achieving sustainable computing and addressing other heterogeneous real-world challenges.
BibTeX
@inproceedings{naug2024sustaindc,
  author = {Avisek Naug and Antonio Guillen and Ricardo Luna and Vineet Gundecha and Desik Rengarajan and Sahand Ghorbanpour and Sajad Mousavi and Ashwin Ramesh Babu and Dejan Markovikj and Lekhapriya Dheeraj Kashyap and Soumyendu Sarkar},
  title = {{SustainDC - Benchmarking for Sustainable Data Center Control}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure