TL;DR
RARE (Renewable energy Aware Resource management) is a Deep Reinforcement Learning (DRL) framework designed to schedule datacenter jobs in environments powered by intermittent renewable energy. By treating the resource pool as a dynamic entity that expands and contracts with power availability, RARE outperforms traditional human-engineered heuristics by up to 25% in total job value.
What Problem Does This Paper Address?
Datacenters are energy gluttons, and while hyperscale providers are committed to going "green," the transition is technically grueling. Unlike the steady hum of the traditional electric grid ("brown energy"), renewable sources like wind and solar are intermittent and volatile.
Traditional job schedulers (like First-Come-First-Serve or Shortest-Job-First) were built for a world where power is a constant. In a green datacenter, the number of available servers might drop at night when the wind dies down or spike at noon when the sun is high. Hand-coding "if-then" rules to handle these fluctuations alongside complex job priorities is expensive, error-prone, and often results in "myopic" decisions that ignore future power predictions. The authors aim to replace these rigid heuristics with an AI agent that can learn to "ride the wave" of available power.
Methods
The authors frame the scheduling problem as a Markov Decision Process (MDP) and solve it using a custom variant of the Deep Actor-Critic framework.
- State Representation: Instead of just a list of numbers, they represent resource availability as a "resource image"—a 2D grid where one axis is time and the other is resource type (CPUs/GPUs). This allows the model to "see" future availability and current allocations.
- The Agent: They use a three-part neural network architecture:
- An Encoder that fuses the resource image and job metadata into a compact vector.
- An Actor that outputs probabilities for scheduling specific jobs.
- A Critic that estimates the long-term value of those actions.
- Learning Paradigms: The paper explores both Online Learning (learning by trial and error in a simulator) and Offline RL (using Critic Regularized Regression to learn from and improve upon existing heuristic datasets).
- Algorithm: The online version is based on Soft Actor-Critic (SAC), which is known for its stability and efficiency in discrete action spaces.
Datasets
The researchers evaluated RARE using a mix of synthetic and real-world data:
- Workloads: They used a synthetic Poisson-process arrival model and the ANL Intrepid HPC workload (a massive log of months of accounting records from Argonne National Laboratory).
- Power Data: Real-world power generation and prediction data (solar and wind) was sourced from the GLEAMM facility at Texas Tech University.
- Benchmarks: RARE was compared against four standard heuristics: Shortest Job First (SJF), Quality of Service (QoS), Highest Value First (HVF), and First Come First Serve (FCFS).
Key Insights and Ideas
- The "Expanding/Contracting" Pool: Unlike most schedulers that assume a fixed number of nodes, RARE treats the resource pool as dynamic. If power drops, resources are marked "black" (unavailable), forcing the agent to prioritize or suspend jobs.
- Planning Horizon Matters: One of the biggest "aha" moments was demonstrating that DRL agents significantly benefit from a longer planning horizon (up to 72 time units). While heuristics are "greedy" and only look at the now, RARE uses power predictions to save resources for high-value jobs expected to arrive later.
- Learning from Experts: The authors showed that RARE doesn't have to start from zero. Using Behavioral Cloning and Offline RL, the agent can mimic a human-designed heuristic (like SJF) and then use its "critic" to find ways to improve upon that baseline.
- State Space Challenges: As datacenters scale from 10 to 300 resources, the performance gap between DRL and heuristics narrows. This suggests that "state space explosion" is a real hurdle for RL in massive clusters.
Key Results
- Performance Boost: In small-scale datacenters (10-50 resources), RARE achieved 18% to 25% higher Total Job Value than the best-performing heuristics.
- Medium-Scale Success: In clusters of 100-300 resources, RARE still outperformed heuristics by 2% to 20%, depending on the workload.
- Real-World Adaptability: When tested with real ANL workloads and GLEAMM power traces, RARE performed 7% to 14% better than traditional policies, proving it can handle the "messiness" of real-world data.
- Offline Improvement: Using Offline RL, the agent improved upon the "Highest Value First" heuristic's performance by 19%.
Limitations and Future Work
- Scalability: The authors acknowledge that as the state space grows (thousands of nodes), the agent struggles to explore all possible actions. They suggest "splitting" the state space into smaller chunks as a future research direction.
- Backfilling: The current framework does not yet support "backfilling"—a common HPC technique where small jobs are tucked into gaps in the schedule. The authors plan to incorporate this in future versions.
- Power Prediction Uncertainty: While the model uses a discount factor ($\gamma$) to handle future uncertainty, the paper largely assumes the power predictions are relatively accurate. Future work could look at how the agent handles "black swan" weather events where predictions fail completely.
Summary updated