Accelerating I/O Performance of SVM on HDFS

Mao Ye, Jun Wang, Jiangling Yin, Xuhong Zhang

Abstract

Hadoop distributed file system (HDFS) is a major distributed file system for commodity clusters and cloud computing. Its extensive scalability and replica fault tolerance scheme makes it well suited for data-intensive application. Due to the tremendous growth of data, many computation-centric applications also become data-intensive. However, they are not optimal on HDFS, which leaves plenty of space for performance optimization. In this paper we ported an MPI-SVM solver, originally developed for HPC environment to the HDFS. We specifically improved the data pre-processing part that requires large amount of I/O operations by a deterministic scheduling method. Our improvement showed a balanced read pattern on each node. The time ratio between the longest process and the shortest process has been reduced by 60%. Also the average read time has significantly reduced by 78%. The data served on each node also showed a small variance in comparison with the originally ported SVM algorithm. We believe that our design avoids the overhead introduced by remote I/O operations, which will be beneficial to many algorithms when coping with large scale of data.
BibTeX
@inproceedings{ye2016accelerating,
  author = {Mao Ye and Jun Wang and Jiangling Yin and Xuhong Zhang},
  title = {{Accelerating I/O Performance of SVM on HDFS}},
  booktitle = {CLUSTER},
  year = {2016},
  doi = {10.1109/cluster.2016.71},
}

Loading related content...

Figure