Enabling Elastic Model Serving with MultiWorld

Myungjin Lee, Akshay Jajoo, Ramana Rao KompellaarXiv 2024 · OpenReview

Abstract: Machine learning models have been exponentially growing in terms of their parameter size over the past few years, leading to the rise of trillion-parameter models. These large models cannot fit into a single GPU and thus require partitioned deployment across GPUs and even hosts. A high-performance collective communication library (CCL) such as NCCL is essential to fully utilize expensive GPU resources. However, CCL is not a great fit for inference. Unlike training, for which a fixed amount of GPU resources is used for fixed workloads, inference workloads can change dynamically over time. Failures at serving time also impact user experience directly. In contrast, workers in a CCL process group share a single fault domain, and the process group cannot grow as workloads increase. To bridge this gap, we propose MultiWorld, which enables fault tolerance and online scaling at the granularity of workers for model serving. Our evaluation showcases that enabling these new functionalities incurs small overheads (1.4-4.3% throughput loss) for most scenarios.

The Challenge of Distributed Inference

As machine learning models cross the trillion-parameter threshold, they can no longer fit on a single GPU. Serving these models requires distributed deployment across multiple GPUs and hosts, typically relying on high-performance Collective Communication Libraries (CCLs) like NVIDIA’s NCCL. However, CCLs were designed for model training, where workloads are static. In model serving (inference), the environment is much more volatile.

Standard CCLs suffer from three primary limitations in serving environments:

  1. Static "Worlds": Once a process group (a "world") is initialized, its size is fixed. Adding or removing workers requires restarting the entire group.
  2. Brittle Fault Tolerance: If a single worker fails, the entire world breaks, leading to service downtime.
  3. Inefficient Scaling: To handle traffic surges, developers must currently deploy an entirely new "world" and replicate the whole model, even if only one specific part of the pipeline is a bottleneck.

The MultiWorld Framework

MultiWorld is built on top of PyTorch to bridge the gap between high-performance CCLs and the need for elastic serving. Its core innovation is allowing a single worker process to join multiple process groups (worlds) dynamically. Instead of one global world, MultiWorld creates smaller, overlapping worlds between specific stages of an inference pipeline.

MultiWorld process group management during normal operation, worker failure, and recovery.

Key technical components include:

  • Multi-World Management: By breaking a large deployment into many small worlds, the fault domain is minimized. If one worker fails, only the worlds it belongs to are affected; others continue working.
  • Non-blocking CCL Operations: To prevent deadlocks, MultiWorld uses asynchronous operations combined with a polling mechanism.
  • Watchdog Daemon: A background thread monitors the "liveness" of all workers via a shared TCPStore. If a worker stops updating its heartbeat, the watchdog alerts the system to clean up the broken worlds.
  • Efficient State Management: The system maintains separate states for different worlds as key-value pairs, avoiding the high performance cost of context switching.

Resilience and Performance

A key advantage of MultiWorld is its ability to isolate failures. In a traditional single-world setup, the termination of one worker causes all other workers to stop. MultiWorld enables "soft-realtime" recovery, where healthy workers continue their progress even after a peer is terminated.

Comparison of sequence progress between single world and MultiWorld during worker termination.

Our evaluation on AWS p3.8xlarge instances (NVIDIA Tesla V100 GPUs) demonstrates that this flexibility comes with minimal cost:

  • Low Overhead: MultiWorld imposes a throughput loss of only 1.4% to 4.3% compared to standard, non-elastic PyTorch setups.
  • Rapid Scaling: Adding a new worker to an active inference job takes only about 20 milliseconds.
  • High Throughput: MultiWorld can saturate a 10 Gbps network link as tensor sizes increase, performing nearly as well as vanilla PyTorch while providing significantly better flexibility.

Conclusion and Future Work

MultiWorld brings much-needed elasticity to distributed model serving. By allowing workers to participate in multiple communication worlds simultaneously, it enables fine-grained fault tolerance and dynamic online scaling for large-scale models. While the current implementation relies on a busy-waiting polling thread that consumes a CPU core, the trade-off is justified by the resulting GPU efficiency and system responsiveness. Future work will focus on the "control plane" logic to automate scaling decisions and further optimize CPU utilization.

Summary updated

Abstract

Machine learning models have been exponentially growing in terms of their parameter size over the past few years. We are now seeing the rise of trillion-parameter models. The large models cannot fit into a single GPU and thus require partitioned deployment across GPUs and even hosts. A high-performance collective communication library (CCL) such as NCCL is essential to fully utilize expensive GPU resources. However, CCL is not a great fit for inference. Unlike training for which a fixed amount of GPU resources is used for fixed workloads (e.g., input datasets), the inference workloads can change dynamically over time. Failures at the serving time can also impact individual user's experiences directly. In contrast, workers in a CCL process group share a single fault domain and the process group cannot grow as the workloads increase. The gap between the unique characteristics of model serving and CCL's nature makes it hard to serve large models elastically. To bridge the gap, we propose MultiWorld that enables fault tolerance and online scaling at the granularity of workers for model serving. Our evaluation showcases that enabling these new functionalities incurs small overheads (1.4-4.3% throughput loss) for most of the scenarios we tested.
BibTeX
@misc{lee2024enabling,
  author = {Myungjin Lee and Akshay Jajoo and Ramana Rao Kompella},
  title = {{Enabling Elastic Model Serving with MultiWorld}},
  year = {2024},
  eprint = {2407.08980},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.08980},
}
Figure