DPDispatcher: Scalable HPC Task Scheduling for AI-Driven Science

Fengbo Yuan, Zhaonan Ding, Yun‐Pei Liu, Kai Cao, Jiahao Fan, Cao Thang Nguyen, Yuzhi Zhang, Haidi Wang, Yixiao Chen, Jiameng Huang, Tongqi Wen, Mingkang Liu, Yifan Li, Yong‐Bin Zhuang, Hao Yu, Ping Tuo, Yaotang Zhang, Yibo Wang, Linfeng Zhang, Handong Wang, Jinzhe ZengJournal of Chemical Information and Modeling 2025 · DOI

Abstract

Artificial intelligence (AI) is reshaping computational science, but AI-driven workflows routinely span heterogeneous tasks executed across diverse high-performance computing (HPC) systems. We introduce DPDispatcher, an open-source Python framework for scalable, fault-tolerant task scheduling in such environments with an emphasis on lightweight submission, automatic retries, and robust resumption. DPDispatcher separates connection and file-staging concerns from scheduler control, supports multiple HPC job managers, and provides both local and secure shell (SSH) backends. DPDispatcher has been adopted by more than ten scientific packages. Representative use cases include active learning for machine-learning potentials, free-energy and thermodynamic integration workflows, large-scale materials screening, and large language model (LLM)-driven agents that launch HPC computations. Across these settings, DPDispatcher reduces operational overhead and error rates while improving portability and automation for reliable, high-throughput scientific computing.
BibTeX
@article{yuan2025dpdispatcher,
  author = {Fengbo Yuan and Zhaonan Ding and Yun‐Pei Liu and Kai Cao and Jiahao Fan and Cao Thang Nguyen and Yuzhi Zhang and Haidi Wang and Yixiao Chen and Jiameng Huang and Tongqi Wen and Mingkang Liu and Yifan Li and Yong‐Bin Zhuang and Hao Yu and Ping Tuo and Yaotang Zhang and Yibo Wang and Linfeng Zhang and Handong Wang and Jinzhe Zeng and Han Wang and Jinzhe Zeng},
  title = {{DPDispatcher: Scalable HPC Task Scheduling for AI-Driven Science}},
  journal = {Journal of Chemical Information and Modeling},
  year = {2025},
  doi = {10.1021/acs.jcim.5c02081},
}

Loading related content...

Figure