A Novel Task Scheduling Approach for Reducing Energy Consumption of MapReduce Cluster

Jie Song, Xuebing Liu, Zhiliang Zhu, Dazhe Zhao, Ge YuIETE 2014 · DOI

TL;DR

This paper tackles the significant energy consumption of MapReduce clusters by proposing a novel task scheduling approach. By focusing on optimizing resource utilization at the component level within nodes, their "Resource Feature Scheduler" (RF Scheduler) aims to minimize "awaiting energy consumption" (AEC) and improve overall energy efficiency and performance.

What Problem Does This Paper Address?

Cloud computing, particularly with data processing frameworks like MapReduce, has become ubiquitous. However, the substantial energy consumption (EC) of these clusters is a growing concern, hindering further development and increasing operational costs. While hardware improvements offer some solutions, software optimization, especially through energy-aware task scheduling, is crucial. Existing MapReduce schedulers (like FIFO, Capacity, and Fair) primarily focus on performance or fairness among users, often neglecting the nuanced resource usage within individual nodes. This can lead to "resource fragmentation" and "awaiting energy consumption" (AEC) – energy wasted when nodes or their components are idle, waiting for other resources or computations. This paper addresses this gap by proposing a scheduling strategy that considers both node-level resource availability and the specific resource demands of different types of tasks.

Methods

The core of the proposed solution lies in two key components:

  • Best Resources Allocation Model (Bram): This model defines an ideal state for resource allocation on a node. It aims to achieve two objectives:
  • Maximize full utilization of each resource: This is achieved by minimizing the sum of differences between allocated and required resource ratios across all resources.
  • Ensure allocated resources meet task requirements: This is done by minimizing the difference between the required and allocated resource vectors for each task. Bram essentially seeks to eliminate "resource fragments" – unallocated idle resources within a node.

  • Resource Feature Scheduling Strategy (RF Scheduler): This is a greedy algorithm designed to achieve the Bram objectives. It works by:

  • Classifying tasks: Tasks are categorized into "task groups" based on their resource requirements (e.g., CPU-intensive, I/O-intensive). This classification forms a hierarchical structure.
  • Matching tasks to resources: The scheduler considers the remaining resources on a node and selects tasks whose resource requirements best match the available resources, aiming to fill resource gaps and avoid fragmentation.
  • Greedy selection: It iteratively selects tasks that best fit the current resource availability, prioritizing those that contribute to minimizing AEC.
  • Handling remaining resources: Any leftover resources that are too small for a new task are allocated to already scheduled tasks to further reduce fragmentation.

Datasets

Not discussed in the paper. The experiments were conducted on a custom Hadoop MapReduce cluster.

Key Insights and Ideas

  • Awaiting Energy Consumption (AEC) is a key target: The paper identifies AEC as a significant source of energy waste in MapReduce clusters, stemming from both node-level and component-level idleness.
  • Component-level optimization is crucial: Unlike previous approaches that focused on node-level resource allocation, this work emphasizes optimizing resource usage within individual nodes by considering the interplay of different resource-intensive tasks.
  • Task type matters: Recognizing that different tasks have distinct resource demands (e.g., CPU vs. I/O intensive) is fundamental. Scheduling complementary tasks together can lead to better resource utilization and reduced AEC.
  • Greedy approach can be effective: While not guaranteeing a globally optimal solution, the RF scheduler's greedy heuristic demonstrates a practical and efficient way to achieve locally optimal solutions that approximate global optima.
  • Performance and energy efficiency are linked: The study suggests that improving energy efficiency through reduced AEC can directly lead to better overall performance by minimizing idle waiting times.

Key Results

The experimental results, conducted on a Hadoop MapReduce cluster with WordCount and Sort jobs, showed that the RF scheduler:

  • Reduced total time consumption: The RF scheduler completed test cases significantly faster than FIFO and Capacity schedulers. For example, in Case C, the RF scheduler took about 150s to reach peak memory usage, while FIFO and Capacity took 300s.
  • Reduced energy consumption: The energy consumed by the RF scheduler was consistently lower than the other schedulers, aligning with the performance improvements.
  • Improved resource utilization: The RF scheduler demonstrated better resource usage by assigning multiple, different resource-intensive tasks to nodes, leading to smaller AEC and less resource fragmentation. Figure 2 visually illustrates how the RF scheduler aims to fill resource gaps.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the RF scheduler is a greedy algorithm and does not guarantee a globally optimal solution.
  • The "resource vector" (Ωu) for tasks is assumed to be known or predictable. The paper does not detail the mechanism for deducing this precise value, relying on prior work or custom experience.
  • The current resource model is capacity-based, and the authors suggest exploring non-capacity resource models.
  • Future Work:
    • Focus on developing more sophisticated resource models and scheduling algorithms, particularly exploring non-capacity resource models.
  • Investigate how to quantify task resource requirements more precisely, as this is noted as a challenging aspect.

Summary updated

Abstract

In recent years, data processing technologies like MapReduce in cloud computing have been widely used. However, the problem of high energy consumption (EC) is an obstacle for further development of cloud computing. We believe that EC could be reduced not only by hardware reinforcement but also by software improvement, especially, by energy-aware task scheduling. In this paper, the Best resources assignment model (Bram) is proposed. On this basis, a scheduling approach, concerning resource usage of nodes and different kinds of resource intensive tasks running on them, is proposed to avoid resource idleness, and then reduce energy consumption. The experimental results show that the new method could reduce the energy consumption of MapReduce cluster.
BibTeX
@inproceedings{song2014novel,
  author = {Jie Song and Xuebing Liu and Zhiliang Zhu and Dazhe Zhao and Ge Yu},
  title = {{A Novel Task Scheduling Approach for Reducing Energy Consumption of MapReduce Cluster}},
  booktitle = {IETE},
  year = {2014},
  doi = {10.1080/02564602.2014.890846},
}
Figure