Titan: a scheduler for foundation model fine-tuning workloads

Wei Gao, Peng Sun, Yonggang Wen, Tianwei Zhang

Abstract

The recent breakthrough of foundation model (FM) research raises a new trend to acquire efficient DL models by fine-tuning FMs with low-resource datasets. Current GPU clusters are mainly established to develop DL models by training from scratch. How to tailor a GPU cluster scheduler for FM fine-tuning workloads is still not explored. We propose Titan, a scheduler to improve the efficiency of FM fine-tuning workloads based on their three distinct features. (1) It takes full advantage of the fixed model structure to estimate the job duration accurately and configure the fine-tuning workload efficiently. (2) The multi-task adaptivity of FMs enables multiple fine-tuning workloads to share the same model parameters, which can significantly reduce the GPU resource consumption. (3) It considers the pipeline parallelism of FM fine-tuning workloads and concurrently executes the parameter transmission and gradient computation to hide the overhead of context switch. Preliminary simulation result validates the efficiency of Titan.
BibTeX
@inproceedings{gao2022titan,
  author = {Wei Gao and Peng Sun and Yonggang Wen and Tianwei Zhang},
  title = {{Titan: a scheduler for foundation model fine-tuning workloads}},
  booktitle = {SoCC},
  year = {2022},
  doi = {10.1145/3542929.3563460},
}

Loading related content...

Figure