Similarity-Based Node Distance Exploring and Locality-Aware Shuffle Optimization for Hadoop MapReduce

Jihe Wang, Danghui Wang, Meng Zhang, Meikang Qiu, Bing Guo

Abstract

To shorten the networking delay from MapTracker to ReduceTracker has attractive potential to gain high performance shuffle for MapReduce. As the original MapReduce shuffle has no locality-aware feature when assigning reduce-tasks over computing nodes, we plan to present a similarity-based distance in the proposed Cloud Node Space to evaluate distance between two computing nodes in data center. Then, we implement a centralized and statistic-based locating service prior networking shuffle to place reduce-tasks near their corresponding data. Experimental results show that, comparing with Hadoop version, this service can achieve 2.3X speedup on shuffle time and bandwidth budget decreases by 60%.
BibTeX
@inproceedings{wang2017similarity,
  author = {Jihe Wang and Danghui Wang and Meng Zhang and Meikang Qiu and Bing Guo},
  title = {{Similarity-Based Node Distance Exploring and Locality-Aware Shuffle Optimization for Hadoop MapReduce}},
  booktitle = {SmartCloud},
  year = {2017},
  doi = {10.1109/smartcloud.2017.23},
}

Loading related content...

Figure