Abstract
The exponential growth in demand for digital services drives massive datacenter energy consumption and negative environmental impacts. Promoting sustainable solutions to pressing energy and digital infrastructure challenges is crucial. Several hyperscale cloud providers have announced plans to power their datacenters using renewable energy. However, integrating renewables to power the datacenters is challenging because the power generation is intermittent, necessitating approaches to tackle power supply variability. Hand engineering domain-specific heuristics-based schedulers to meet specific objective functions in such complex dynamic green datacenter environments is time-consuming, expensive, and requires extensive tuning by domain experts. The green datacenters need smart systems and system software to employ multiple renewable energy sources (wind and solar) by intelligently adapting computing to renewable energy generation. We present RARE (Renewable energy Aware REsource management), a Deep Reinforcement Learning (DRL) job scheduler that automatically learns effective job scheduling policies while continually adapting to datacenters' complex dynamic environment. The resulting DRL scheduler performs better than heuristic scheduling policies with different workloads and adapts to the intermittent power supply from renewables. We demonstrate DRL scheduler system design parameters that, when tuned correctly, produce better performance. Finally, we demonstrate that the DRL scheduler can learn from and improve upon existing heuristic policies using Offline Learning.
Introduction
Datacenters are significant energy consumers, and while hyperscale providers are committed to transitioning to sustainable energy, the technical implementation is grueling. Unlike the steady supply of the traditional electric grid ("brown energy"), renewable sources like wind and solar are inherently intermittent and volatile.
Traditional job schedulers, such as First-Come-First-Serve (FCFS) or Shortest-Job-First (SJF), were designed for environments where power is a constant. In a green datacenter, the number of available servers may drop at night when the wind dies down or spike at noon when solar generation peaks. Hand-coding "if-then" rules to handle these fluctuations alongside complex job priorities is expensive, error-prone, and often results in "myopic" decisions that ignore future power predictions. The RARE framework aims to replace these rigid heuristics with an AI agent capable of learning to optimize job scheduling based on available power availability.
Methodology
The RARE framework frames the scheduling problem as a Markov Decision Process (MDP) and utilizes a custom variant of the Deep Actor-Critic framework to solve it.
State Representation
To allow the model to perceive future availability and current allocations, resource availability is represented as a "resource image." This is a 2D grid where one axis represents time and the other represents resource types (e.g., CPUs/GPUs). This spatial representation enables the agent to "see" the impact of its decisions over a planning horizon.
Agent Architecture
The agent utilizes a three-part neural network architecture:
- Encoder: Fuses the resource image and job metadata into a compact vector representation.
- Actor: Outputs probabilities for scheduling specific jobs from the queue.
- Critic: Estimates the long-term value of those actions to guide the learning process.
Learning Paradigms
The research explores two primary learning modes:
- Online Learning: Based on Soft Actor-Critic (SAC), the agent learns through trial and error within a simulated environment, optimizing for stability and efficiency.
- Offline RL: Using Critic Regularized Regression (CRR), the agent learns from existing heuristic datasets, allowing it to mimic and eventually surpass human-designed policies.
Experimental Setup
RARE was evaluated using a combination of synthetic and real-world data:
- Workloads: Synthetic Poisson-process arrival models and the ANL Intrepid HPC workload (accounting records from Argonne National Laboratory).
- Power Data: Real-world solar and wind generation and prediction data from the GLEAMM facility at Texas Tech University.
- Benchmarks: RARE was compared against four standard heuristics: Shortest Job First (SJF), Quality of Service (QoS), Highest Value First (HVF), and First Come First Serve (FCFS).
Key Insights
The Dynamic Resource Pool
Unlike traditional schedulers that assume a fixed number of nodes, RARE treats the resource pool as an expanding and contracting entity. When power generation drops, resources are marked as unavailable, forcing the agent to prioritize or suspend jobs dynamically.
Planning Horizon and Future Awareness
A critical finding was that DRL agents benefit significantly from a longer planning horizon (up to 72 time units). While heuristics are typically "greedy"—focusing only on immediate rewards—RARE uses power predictions to reserve resources for high-value jobs expected to arrive in the future.
Learning from Experts
RARE does not need to start from a blank slate. Through Behavioral Cloning and Offline RL, the agent can first learn to mimic a human-designed heuristic (like SJF) and then use its critic component to identify and implement improvements over that baseline.
Results
- Performance Gains: In small-scale datacenters (10-50 resources), RARE achieved 18% to 25% higher Total Job Value than the best-performing heuristics.
- Scalability: In clusters of 100-300 resources, RARE maintained a performance lead of 2% to 20% over traditional methods.
- Real-World Application: When tested with real ANL workloads and GLEAMM power traces, RARE performed 7% to 14% better than traditional policies, demonstrating robustness against real-world data volatility.
- Offline Optimization: Using Offline RL, the agent improved upon the "Highest Value First" heuristic's performance by 19%.
Limitations and Future Work
While RARE shows significant promise, scalability remains a challenge. As datacenters grow to thousands of nodes, the state space explosion makes exploration difficult. Future research will focus on "splitting" the state space into manageable segments. Additionally, the authors plan to incorporate "backfilling"—a technique to tuck small jobs into schedule gaps—and investigate how agents handle extreme "black swan" weather events where power predictions fail entirely.
Summary updated