Towards Self-Tuning Parameter Servers

Chris Liu, Pengfei Zhang, Bo Tang, Hang Shen, Lei Zhu, Ziliang Lai, Eric LoOpenReview 2018 · DOI · OpenReview

Abstract

Recent years, many applications have been driven advances by the use of Machine Learning (ML). Nowadays, it is common to see industrial-strength machine learning jobs that involve millions of model parameters, terabytes of training data, and weeks of training. Good efficiency, i.e., fast completion time of running a specific ML training job, therefore, is a key feature of a successful ML system. While the completion time of a long-running ML job is determined by the time required to reach model convergence, that is also largely influenced by the values of various system settings. In this paper, we contribute techniques towards building self-tuning parameter servers. Parameter Server (PS) is a popular system architecture for large-scale machine learning systems; and by self-tuning we mean while a long-running ML job is iteratively training the expert-suggested model, the system is also iteratively learning which system setting is more efficient for that job and applies it online. Our techniques are general enough to various PS-style ML systems. Experiments on TensorFlow show that our techniques can reduce the completion times of a variety of long-running TensorFlow jobs from 1.4× to 18×.

Introduction

Training industrial-strength machine learning models is a massive undertaking, often requiring weeks of computation on large clusters. Most distributed systems utilize a Parameter Server (PS) architecture, where "workers" compute gradients and "servers" maintain the global model state. However, the performance of these systems is highly sensitive to various "knobs"—system parameters such as the server-to-worker ratio, thread affinity, and memory optimization flags.

Currently, these settings are manually configured by users, which is problematic for two main reasons. First, the relationship between these knobs and training speed is non-monotonic and extremely complex. As shown in Figure 1, the response surface for completion time is far from intuitive, making manual optimization nearly impossible. Second, manual settings are static; a configuration optimal at the start of training may become inefficient as the model converges.

The complex response surface of completion time relative to system parameters like thread parallelism and worker count.

The TensorFlowOnline Framework

To address these challenges, we introduce TensorFlowOnline, a framework that transforms the PS architecture into a self-tuning system. The system operates while the model is training, iteratively discovering and applying optimal hardware settings.

System Architecture

The framework consists of several key components:

  • Instrumentation: Monitors the training process to collect execution metrics.
  • Tuning Manager: Uses Bayesian Optimization (BO) with Gaussian Processes to model the relationship between system knobs and the remaining completion time. It balances exploring new configurations with exploiting known high-performing ones.
  • Repository: Stores historical metrics to inform the tuning process.
  • ML System Back-end: Executes the training job using the parameters provided by the Tuning Manager.
The architecture of the self-tuning framework, showing the interaction between the front-end, instrumentation, tuning manager, and back-end.

Key Techniques

  1. Online Progress Estimation: To optimize for total completion time, the system must estimate how much time is left. We developed a method to estimate statistical progress—the number of iterations needed to reach convergence—by adapting theoretical convergence bounds into a live estimation function.
  2. On-Demand Model Relocation (ODMR): Traditional reconfiguration requires stopping the job and checkpointing to disk. ODMR allows the system to reconfigure on the fly by moving model parameters only when a worker requests them, significantly reducing downtime.

Experimental Evaluation

We evaluated TensorFlowOnline using standard benchmarks, including KDD12 (Logistic Regression), CRITEO (SVM), CIFAR10 (CNN), and ImageNet8 (AlexNet).

Results and Insights

The experiments revealed a fundamental "Efficiency Tug-of-War": total training time is the product of hardware efficiency (seconds per iteration) and statistical efficiency (iterations to convergence). A configuration that speeds up individual iterations might actually increase the total time if it slows down convergence due to noise in asynchronous updates.

TensorFlowOnline demonstrated significant performance gains:

  • Speedups: It achieved a 1.4× to 2.5× speedup over average human-selected settings.
  • Robustness: Compared to poor manual choices, the system was 6× to 18× faster, effectively avoiding performance "disasters."
  • Reconfiguration Efficiency: The ODMR technique reduced reconfiguration costs by 400% to 760% compared to standard checkpoint-and-restart methods.

Conclusion and Future Work

TensorFlowOnline successfully automates the complex task of tuning parameter servers. While the current progress estimation relies on convex assumptions, it serves as an effective heuristic for non-convex deep learning models. Future work will explore more specialized estimation functions for deep learning, address the "cold start" problem through transfer learning, and extend the framework to heterogeneous hardware environments.

Summary updated

Abstract

Recent years, many applications have been driven advances by the use of Machine Learning (ML). Nowadays, it is common to see industrial-strength machine learning jobs that involve millions of model parameters, terabytes of training data, and weeks of training. Good efficiency, i.e., fast completion time of running a specific ML job, therefore, is a key feature of a successful ML system. While the completion time of a long-running ML job is determined by the time required to reach model convergence, practically that is also largely influenced by the values of various system settings. In this paper, we contribute techniques towards building self-tuning parameter servers. Parameter Server (PS) is a popular system architecture for large-scale machine learning systems; and by self-tuning we mean while a long-running ML job is iteratively training the expert-suggested model, the system is also iteratively learning which system setting is more efficient for that job and applies it online. While our techniques are general enough to various PS-style ML systems, we have prototyped our techniques on top of TensorFlow. Experiments show that our techniques can reduce the completion times of a variety of long-running TensorFlow jobs from 1.4x to 18x.
BibTeX
@misc{liu2018towards,
  author = {Chris Liu and Pengfei Zhang and Bo Tang and Hang Shen and Lei Zhu and Ziliang Lai and Eric Lo},
  title = {{Towards Self-Tuning Parameter Servers}},
  year = {2018},
  eprint = {1810.02935},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.02935},
  howpublished = {OpenReview},
}
Figure