DPDispatcher: Scalable HPC Task Scheduling for AI-Driven Science
Lists
Only you see this
Abstract
Artificial intelligence (AI) is reshaping computational science, but AI-driven workflows routinely span heterogeneous tasks executed across diverse high-performance computing (HPC) systems. We introduce DPDispatcher, an open-source Python framework for scalable, fault-tolerant task scheduling in such environments with an emphasis on lightweight submission, automatic retries, and robust resumption. DPDispatcher separates connection and file-staging concerns from scheduler control, supports multiple HPC job managers, and provides both local and secure shell (SSH) backends. DPDispatcher has been adopted by more than ten scientific packages. Representative use cases include active learning for machine-learning potentials, free-energy and thermodynamic integration workflows, large-scale materials screening, and large language model (LLM)-driven agents that launch HPC computations. Across these settings, DPDispatcher reduces operational overhead and error rates while improving portability and automation for reliable, high-throughput scientific computing.
BibTeX
@article{yuan2025dpdispatcher,
author = {Fengbo Yuan and Zhaonan Ding and Yun‐Pei Liu and Kai Cao and Jiahao Fan and Cao Thang Nguyen and Yuzhi Zhang and Haidi Wang and Yixiao Chen and Jiameng Huang and Tongqi Wen and Mingkang Liu and Yifan Li and Yong‐Bin Zhuang and Hao Yu and Ping Tuo and Yaotang Zhang and Yibo Wang and Linfeng Zhang and Handong Wang and Jinzhe Zeng and Han Wang and Jinzhe Zeng},
title = {{DPDispatcher: Scalable HPC Task Scheduling for AI-Driven Science}},
journal = {Journal of Chemical Information and Modeling},
year = {2025},
doi = {10.1021/acs.jcim.5c02081},
}Loading related content...