Online Runtime Prediction Method for Distributed Iterative Jobs

Xiaofei Yue, Lan Shi, Yuhai Zhao, Hangxu Ji, Guoren Wang

Abstract

Predicting the runtime of distributed iterative jobs can help reduce the deployment cost of clusters and optimize their resource allocation and scheduling strategies, but the runtime depends on various factors which are difficult to be acquired before execution. In this paper, we propose a generalized online prediction method for the runtime of distributed iterative jobs, which is centered on a series of online machine learning models. The method consists of three phases: 1) estimating the number of iterations for the current iterative job. 2) predicting the runtime metrics of each iteration by an online polynomial regression model. 3) Runtime metrics sequence is analyzed using an LSTM trained with online learning to predict the runtime of each iteration. We conducted experiments on typical Flink iterative jobs, and the experimental results show that our method improves the accuracy by 4.79% compared to the state-of-the-art methods, while for the improvement in accuracy for delta iterative jobs is even more than 15%.
BibTeX
@inproceedings{yue2021online,
  author = {Xiaofei Yue and Lan Shi and Yuhai Zhao and Hangxu Ji and Guoren Wang},
  title = {{Online Runtime Prediction Method for Distributed Iterative Jobs}},
  booktitle = {WISA},
  year = {2021},
  doi = {10.1007/978-3-030-87571-8\_14},
}

Loading related content...

Figure