TL;DR
RHAPSODY is a multi-runtime middleware designed to execute "hybrid" workflows that combine massive HPC simulations with real-time AI inference and autonomous agents. By composing existing runtimes rather than replacing them, it enables these diverse workloads to coexist efficiently in a single supercomputer allocation with minimal overhead and near-linear scaling.
What Problem Does This Paper Address?
Modern scientific discovery is moving away from "pure" simulations. Researchers now use hybrid AI-HPC workflows: for example, a physics simulation might pause to call an AI model (inference) to predict the next step, or an autonomous AI "agent" might steer thousands of small simulations based on real-time results.
The problem is that traditional High-Performance Computing (HPC) systems and modern AI frameworks speak different languages. HPC runtimes are built for massive, long-running MPI (Message Passing Interface) jobs. AI frameworks are built for persistent services, high-throughput inference, and fine-grained Python tasks. Trying to run both simultaneously often leads to "resource silos," where hardware sits idle during hand-offs, or massive performance hits due to the high latency of moving data through the filesystem between the AI and HPC components.
Methods
The authors introduce RHAPSODY, a middleware layer that acts as a "manager of managers." Its core philosophy is composition: instead of building a new scheduler from scratch, it coordinates specialized backends (like Dragon for distributed Python and Flux for HPC tasks).
Key technical features include:
- Uniform Abstractions: It treats MPI executables, Python functions, and long-running AI services (like vLLM) as the same basic unit: a "task."
- Layered Architecture: A public API handles task descriptions and policies, while a middleware layer manages a global view of resources (CPUs, GPUs, and active services).
- Service Lifecycle Management: RHAPSODY treats AI models as first-class entities. It handles their instantiation, checks when they are "ready" to receive requests, and manages endpoint discovery so other tasks can find them.
- Memory-Based Coupling: To avoid the "filesystem bottleneck," it integrates with tools like SmartRedis to exchange data between AI and HPC tasks directly in memory.
Datasets
Rather than using a single static dataset, the authors evaluated RHAPSODY using three representative "scientific campaign" patterns:
- IMPECCABLE (Heterogeneous Workloads): A drug-discovery pipeline mixing CPU-heavy preprocessing, GPU-accelerated ML docking, and large MPI simulations.
- LUCID/SPHERICAL (Inference at Scale): Workflows that use Large Language Models (LLMs) to screen scientific literature or design antigens, requiring high-volume, concurrent inference requests.
- IMPRESS (Agentic Workflows): An AI planning agent that dynamically selects and orders protein structure simulations based on intermediate results.
Key Insights and Ideas
- Composition over Replacement: The most significant insight is that we don't need a "one size fits all" runtime. By providing a thin coordination layer over existing, battle-tested runtimes, RHAPSODY achieves flexibility without the "re-inventing the wheel" performance tax.
- Heterogeneity Width (HW): The authors propose a metric called "Heterogeneity Width"—the number of distinct task types running concurrently. They found that RHAPSODY allows HW to grow naturally with scale; as you add more nodes, the system can support more complex, diverse science rather than just "more of the same."
- The "Agentic" Lag: In workflows where an AI agent makes decisions, the time between the decision and the execution (the "realization rate") is critical. RHAPSODY demonstrates "bounded lag," meaning the system doesn't get bogged down even when the AI agent generates a burst of new tasks.
Key Results
- Minimal Overhead: RHAPSODY introduces only 100–300 $\mu$s of overhead per task, making it suitable for workflows with thousands of fine-grained operations.
- Near-Linear Scaling: For high-throughput inference (using vLLM), the system achieved near-linear scaling, reaching $3.5 \times 10^5$ tokens per second across 8 nodes.
- Efficiency Gains: Using memory-based data exchange (via Redis) instead of traditional filesystem-based exchange resulted in up to a 50% reduction in total runtime for tightly coupled workflows.
- GPU Utilization: The system maintained consistently high GPU utilization (90–95%) even while managing complex, asynchronous request streams.
Limitations and Future Work
- Backend Dependence: While designed to be general, the current evaluation is heavily centered on the Dragon and vLLM backends. Future work could explore how it handles other emerging AI runtimes.
- Static Partitioning: Currently, the system often partitions nodes between different backends. More dynamic, "malleable" resource sharing—where a node might switch from running an MPI rank to an inference service on the fly—is a potential area for optimization.
- Deployment: RHAPSODY is currently a research prototype; the authors aim to deploy it on the American Science Cloud to support production-level scientific workflows.
Summary updated