Titan: a scheduler for foundation model fine-tuning workloads
Lists
Only you see this
Abstract
The recent breakthrough of foundation model (FM) research raises a new trend to acquire efficient DL models by fine-tuning FMs with low-resource datasets. Current GPU clusters are mainly established to develop DL models by training from scratch. How to tailor a GPU cluster scheduler for FM fine-tuning workloads is still not explored. We propose Titan, a scheduler to improve the efficiency of FM fine-tuning workloads based on their three distinct features. (1) It takes full advantage of the fixed model structure to estimate the job duration accurately and configure the fine-tuning workload efficiently. (2) The multi-task adaptivity of FMs enables multiple fine-tuning workloads to share the same model parameters, which can significantly reduce the GPU resource consumption. (3) It considers the pipeline parallelism of FM fine-tuning workloads and concurrently executes the parameter transmission and gradient computation to hide the overhead of context switch. Preliminary simulation result validates the efficiency of Titan.
BibTeX
@inproceedings{gao2022titan,
author = {Wei Gao and Peng Sun and Yonggang Wen and Tianwei Zhang},
title = {{Titan: a scheduler for foundation model fine-tuning workloads}},
booktitle = {SoCC},
year = {2022},
doi = {10.1145/3542929.3563460},
}Loading related content...