RFHOC: A Random-Forest Approach to Auto-Tuning Hadoop's Configuration
Lists
Only you see this
Abstract
Hadoop is a widely-used implementation framework of the MapReduce programming model for large-scale data processing. Hadoop performance however is significantly affected by the settings of the Hadoop configuration parameters. Unfortunately, manually tuning these parameters is very time-consuming, if at all practical. This paper proposes an approach, called RFHOC, to automatically tune the Hadoop configuration parameters for optimized performance for a given application running on a given cluster. RFHOC constructs two ensembles of performance models using a random-forest approach for the map and reduce stage respectively. Leveraging these models, RFHOC employs a genetic algorithm to automatically search the Hadoop configuration space. The evaluation of RFHOC using five typical Hadoop programs, each with five different input data sets, shows that it achieves a performance speedup by a factor of 2.11 <inline-formula><tex-math notation="LaTeX">$\times$</tex-math></inline-formula> on average and up to 7.4 <inline-formula><tex-math notation="LaTeX">$\times$</tex-math></inline-formula> over the recently proposed cost-based optimization (CBO) approach. In addition, RFHOC's performance benefit increases with input data set size.
BibTeX
@article{bei2015rfhoc,
author = {Zhendong Bei and Zhibin Yu and Huiling Zhang and Wen Xiong and Chengzhong Xu and Lieven Eeckhout and Shengzhong Feng},
title = {{RFHOC: A Random-Forest Approach to Auto-Tuning Hadoop's Configuration}},
journal = {IEEE Transactions on Parallel and Distributed Systems},
year = {2015},
doi = {10.1109/tpds.2015.2449299},
}Loading related content...