TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism
Lists
Only you see this
Abstract
Effective parallelization strategies are crucial for the performance of distributed deep neural network (DNN) training. Recently, several methods have been proposed to search parallelization strategies but they all optimize a single objective (e.g., execution time, memory consumption) and produce only one strategy. We propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Frontier Tracking</i> (FT), an efficient algorithm that finds <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a set of Pareto-optimal parallelization strategies</i> to explore the best trade-off among different objectives. FT can minimize the memory consumption when the number of devices is limited and fully utilize additional resources to reduce the execution time. Based on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FT</i> , we develop a user-friendly system, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TensorOpt</i> , which allows users to run their distributed DNN training jobs without caring the details about searching and coding parallelization strategies. Experimental results show that TensorOpt is more flexible in adapting to resource availability compared with existing frameworks.
BibTeX
@article{cai2022tensoropt,
author = {Zhenkun Cai and Xiao Yan and Kaihao Ma and Yidi Wu and Yuzhen Huang and James Cheng and Teng Su and Fan Yu},
title = {{TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism}},
journal = {IEEE Trans. Parallel Distributed Syst.},
year = {2022},
url = {https://openreview.net/forum?id=30WbTpgVQSU},
}Loading related content...