TL;DR
This paper tackles the challenge of efficiently integrating intermittent renewable energy sources (solar and wind) into data centers. They propose a novel job scheduling system that uses short-term green energy predictions to dynamically adjust the workload, significantly improving green energy utilization and reducing job cancellations compared to relying on instantaneous energy availability.
What Problem Does This Paper Address?
Data centers are massive energy consumers, contributing significantly to global greenhouse gas emissions. While renewable energy sources like solar and wind offer a greener alternative, their inherent variability poses a major challenge. Traditional data center workloads, especially time-sensitive services, often require a stable power supply, making it difficult to rely on fluctuating green energy. Previous approaches either canceled jobs when green energy dipped or only used it instantaneously, leading to wasted energy and job interruptions. This paper aims to bridge this gap by developing a predictive scheduling system that can intelligently leverage short-term green energy forecasts to optimize job execution and maximize the use of renewables without compromising service-level agreements.
Methods
The core of this work is a new adaptive data center job scheduler that integrates short-term (30-minute interval) predictions of solar and wind energy production.
- Green Energy Prediction:
- Solar: They adapted the Weather-Conditioned Moving Average (WCMA) algorithm, achieving a mean error of 9.6% for a 30-minute prediction window, even in variable weather conditions.
- Wind: A novel, low-overhead predictor was developed using weighted nearest-neighbor (NN) tables. This method utilizes wind speed and direction data to generate power curves, adapting to seasonal changes by weighting recent results. It achieved a mean error of 17.2% for a 30-minute prediction interval, outperforming some time-series models at a lower computational cost.
- Job Scheduling: The scheduler uses two job queues: one for time-sensitive web services and another for batch jobs (like MapReduce).
- Service Jobs: These are prioritized and guaranteed enough brown energy to meet their response time requirements.
- Batch Jobs: The scheduler dynamically adjusts the number of batch job slots based on the predicted green energy availability. If predicted energy is high, more slots are allocated; if low, some jobs are deallocated to conserve energy or ensure service jobs are met. This approach aims to reduce job cancellations and re-executions.
Datasets
- Solar Energy Data: Measured data from a solar farm at the University of California, San Diego.
- Wind Energy Data: Power output data from a wind farm provided by the National Renewable Energy Laboratory (NREL) and meteorological data from the National Climatic Data Center (NCDC).
- Workload Data:
- Service Jobs: Simulated using the Rubis benchmark, with interarrival times modeled by a lognormal distribution.
- Batch Jobs: Simulated using the MapReduce framework (specifically Hadoop), with job arrival times modeled by a lognormal distribution.
The paper also mentions using measurements from a testbed of Intel Nehalem servers running a mix of Rubis and MapReduce workloads to validate their simulation models.
Key Insights and Ideas
- Prediction is Key: The most significant insight is that short-term prediction of green energy availability is far more effective than reacting to instantaneous supply. This allows for proactive resource allocation and workload management.
- Balancing Act: The scheduler successfully balances the needs of latency-sensitive service jobs with the throughput-oriented batch jobs, ensuring that neither is unduly sacrificed.
- Reducing Waste: By predicting energy availability, the system can avoid starting jobs that are likely to be interrupted, thus reducing wasted energy and the overhead of re-execution.
- Data Center as a Computer: The paper implicitly reinforces the idea of viewing a data center as a cohesive computing system where energy availability is a first-class resource to be managed.
Key Results
The proposed predictive scheduler demonstrated significant improvements over an instantaneous green energy scheduler:
- Green Energy Usage Efficiency: Increased by up to 3x.
- Number of Terminated Jobs: Reduced by up to 7.7x.
- MapReduce Tasks Executed with Green Energy: Increased by 2x.
- Brown Energy Reduction: When supplementing with brown energy to ensure all jobs complete, the predictive scheduler required 7x less additional brown energy (0.64 kWh vs. 4.6 kWh per container) compared to the instantaneous approach.
- MapReduce Task Completion Times: Decreased by an average of 20% compared to a baseline using only brown energy.
- Service QoS: Response time requirements for web services were consistently met, with QoS ratios remaining well below 1.
Limitations and Future Work
- Simulation-Based Evaluation: While the simulation platform was validated against real measurements, the primary evaluation is through simulation.
- Specific Prediction Interval: The 30-minute prediction interval was chosen based on MapReduce job runtimes. Its effectiveness for other job types or shorter intervals might vary.
- Future Work: The authors suggest exploring more sophisticated prediction models and investigating the impact of different energy storage solutions in conjunction with their predictive scheduling. They also highlight the potential for extending this approach to other types of green energy sources and data center architectures.
Summary updated