Scalable HPC & AI infrastructure for COVID-19 therapeutics

Hyungro Lee, André Merzky, Li Tan, Mikhail Titov, Matteo Turilli, Dario Alfè, Agastya Bhati, Alex Brace, Austin Clyde, Peter V. Coveney, Heng Ma, Arvind Ramanathan, Rick Stevens, Anda Trifan, Hubertus Van Dam, Shunzhou Wan, Sean R. Wilkinson, Shantenu JhaPASC 2021 · DOI

Abstract

This paper presents a scalable computational infrastructure that integrates artificial intelligence (AI) with physics-based simulations to accelerate drug discovery for COVID-19. By deploying heterogeneous workflows across multiple global supercomputers, the research team successfully screened over 4.2 billion molecules, identifying over 1,000 promising compounds for experimental validation. The work highlights the shift from pure simulation to AI-steered data generation in high-performance computing (HPC).

Introduction: The Massive Search Space of Drug Discovery

In silico (computational) drug discovery represents a monumental search problem. The chemical space of potential drug-like molecules is estimated at $10^{68}$ compounds. Navigating this space to find a molecule that effectively binds to a viral target, such as a SARS-CoV-2 protein, requires modeling across multiple physical length- and timescales.

No single computational method currently offers the necessary balance of speed and accuracy. Fast docking methods can screen billions of molecules but lack precision, while highly accurate molecular dynamics (MD) simulations are computationally prohibitive at scale. Consequently, researchers must orchestrate multi-stage workflows that chain fast filters, machine learning (ML) surrogates, and high-fidelity physics simulations. Managing these workflows across diverse, heterogeneous HPC systems like Summit, Frontera, and Lassen without significant overhead or system failure is a critical systems engineering challenge.

Iterative Workflow Architecture

The campaign utilized two iterative loops—Hit-to-Lead and Lead Optimization—powered by four distinct workflows (WFs).

The iterative drug discovery pipeline showing the Hit-to-Lead and Lead Optimization loops.
  1. WF1 (Ensemble Docking): Rapidly screens 3D molecular structures against protein targets using OpenEye (CPU) and AutoDock-GPU.
  2. WF2 (ML-Driven Enhanced Sampling): Employs DeepDriveMD, coupling MD simulations with a Convolutional Variational Autoencoder (CVAE). The ML model clusters trajectories and steers simulations toward unexplored conformational states, avoiding computational dead ends.
  3. WF3 & WF4 (Free Energy Estimation): Uses ESMACS (coarse-grained, GPU-bound via OpenMM) and TIES (fine-grained, CPU-bound via NAMD) to calculate binding affinities with high precision.

Systems Engineering for Heterogeneous HPC

To execute these workflows at scale, the team employed RADICAL-Cybertools (RCT). A primary innovation was the implementation of heterogeneous task placement. Modern supercomputer nodes are inherently heterogeneous; for instance, a Summit node contains 42 CPUs and 6 GPUs. Running GPU-only tasks leaves the CPUs idle, creating "dark space." By merging CPU-heavy (TIES) and GPU-heavy (ESMACS) tasks into a single hybrid workflow, the team ran both concurrently on the same nodes, drastically improving resource utilization.

Furthermore, the team addressed the fragility of standard HPC job launchers. At massive scales, system-level resource managers (like PRRTE/PMIx) often fail. The researchers bypassed these bottlenecks using novel execution modes, including the RAPTOR master/worker overlay and multi-Distributed Virtual Machines (multi-DVM), which provided a more robust scheduling layer.

Performance and Scientific Impact

The campaign utilized 2.5 million node-hours across five major supercomputing centers (Summit, Lassen, Frontera, Theta, and SuperMUC-NG). Key performance metrics include:

  • Throughput: A peak docking rate of 40 million docks per hour, with sustained rates between 11 and 27 million depending on the system and ligand database.
  • Efficiency: The hybrid workflow scaled to 22,640 concurrent tasks on 128 Summit nodes, with middleware overhead kept remarkably low (3.8% to 11.5%).
  • Scientific Discovery: Over 4.2 billion molecules from libraries like Orderable-zinc-db-enaHLL were screened against various SARS-CoV-2 targets, such as the ADRP protein, leading to the experimental validation of over 1,000 compounds.

Scaling Bottlenecks and Future Work

Despite the success, significant challenges remain. One primary issue is the long-tail distribution of task durations.

Histograms of docking times illustrating the significant 'long-tail' distribution that complicates load balancing.

As shown in the docking time histograms, task durations can range from 0.1 seconds to nearly an hour. This variance creates severe load-balancing issues, where some nodes remain idle while others are bogged down by "heavy" tasks. Additionally, pushing to extreme scales (2,000+ nodes) revealed persistent system software fragility, including connectivity losses between DVMs and executors.

Future work focuses on deeper integration with the Flux resource management framework. Early tests with Flux demonstrated 88% utilization and zero failures at smaller scales, suggesting a path toward even more robust production runs at the exascale level.

Summary updated

BibTeX
@inproceedings{lee2021scalable,
  author = {Hyungro Lee and André Merzky and Li Tan and Mikhail Titov and Matteo Turilli and Dario Alfè and Agastya Bhati and Alex Brace and Austin Clyde and Peter V. Coveney and Heng Ma and Arvind Ramanathan and Rick Stevens and Anda Trifan and Hubertus Van Dam and Shunzhou Wan and Sean R. Wilkinson and Shantenu Jha},
  title = {{Scalable HPC \& AI infrastructure for COVID-19 therapeutics}},
  booktitle = {PASC},
  year = {2021},
  doi = {10.1145/3468267.3470573},
}
Figure