BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching

Dingyan Zhang, Haotian Wang, Yang Liu, Xingda Wei, Yizhou Shan, Rong Chen, Haibo ChenOSDI 2025

TL;DR

This paper introduces BlitzScale, a novel system for autoscaling large language models (LLMs) that dramatically speeds up the process by leveraging the compute network for parameter loading and enabling "live" scaling. This allows LLM services to handle sudden bursts of traffic with significantly reduced latency and improved resource utilization compared to existing methods.

What Problem Does This Paper Address?

Serving large language models (LLMs) as a service (MAAS) faces a critical challenge: autoscaling. When traffic surges, MAAS systems need to quickly provision more instances to avoid performance degradation and meet service level objectives (SLOs). However, existing autoscaling mechanisms are slow because loading massive model parameters (tens to hundreds of gigabytes) onto GPUs is a bottleneck. This "stop-the-world" approach, where new instances are unusable until fully loaded, leads to significant latency spikes during traffic bursts. Current solutions like host caching offer some improvement but are impractical for MAAS environments hosting numerous models, leading to cache misses and still slow scaling.

Methods

BlitzScale tackles this problem with two core innovations:

  • Fast Data Plane with O(1) or No Caching: Instead of relying heavily on host caching, BlitzScale leverages the high-bandwidth, often underutilized, compute network (e.g., RDMA, NVLink) between GPUs and CPUs. It uses a model-aware multicast planner to efficiently distribute model parameters across available resources. This approach minimizes or eliminates the need for extensive host caching, reducing scaling time.
  • Live Autoscaling with Fine-Grained Abstraction: BlitzScale breaks down the scaling process from an instance-level to a layer-level. This allows newly scaled instances to start processing requests even before all model parameters are loaded. By offloading computations layer-by-layer from overloaded instances to partially loaded ones, BlitzScale achieves "live" scaling, improving system throughput incrementally as more layers become available. This is managed through a novel cooperative execution scheme and ZigZag scheduling.

Datasets

The paper evaluates BlitzScale using real-world traces:

  • BurstGPT [71]: A trace representing chat workloads.
  • AzureCode [14]: A trace representing code generation workloads.
  • AzureConv [14]: A trace representing conversational workloads.

These traces were scaled to fit the evaluation clusters while preserving their temporal patterns.

Key Insights and Ideas

  • Compute Network is Underrated for Scaling: The paper demonstrates that the compute network, often underutilized during normal serving, can be effectively repurposed for fast model parameter transfer during autoscaling, significantly outperforming SSD-based loading.
  • Live Scaling is Crucial for Bursty Workloads: Waiting for full parameter loading is a major bottleneck. Enabling instances to contribute to serving as soon as partial parameters are available is key to meeting strict SLOs during traffic spikes.
  • Layer-Level Abstraction Unlocks Live Scaling: By treating model execution and scaling at a finer granularity (layer-by-layer), systems can achieve incremental throughput gains during the scaling process.
  • Model-Aware Multicast is Key: Efficiently planning parameter distribution across a dynamic network topology requires understanding model structures and network capabilities to avoid interference with serving traffic.

Key Results

  • Reduced Tail Latency: BlitzScale achieves up to 94% lower tail latency reductions compared to the state-of-the-art autoscaling system (ServerlessLLM).
  • Faster Time-to-First-Token (TTFT) and Time-Between-Tokens (TBT): BlitzScale offers 47-75% shorter TTFT and up to 94% shorter TBT compared to ServerlessLLM.
  • Improved GPU Utilization: BlitzScale reduces GPU time used for serving by 49% compared to systems without autoscaling (like DistServe and vLLM) while meeting the same service-level agreements.
  • Minimal Host Caching: BlitzScale requires minimal host caching (less than one copy per model) due to its efficient network-based multicast, unlike ServerlessLLM which can "pollute" host cache with many models.
  • Negligible Network Overhead: Despite frequent autoscaling, BlitzScale's additional network usage is negligible.

Limitations and Future Work

  • Scaling Policies: The paper acknowledges that autoscaling policies (when and how to scale) are orthogonal to their work but can significantly impact system efficiency. They suggest this as a future research direction, noting that policies depend heavily on workload characteristics.
  • Instance-Level Scaling: BlitzScale currently focuses on instance-level scaling. Future work could explore scaling experts within Mixture-of-Experts (MoE) models or changing parallel configurations within an instance.
  • LLM Specific Optimizations: While BlitzScale has specializations for LLMs (e.g., PD colocation, decode instance scaling), further optimizations for specific LLM architectures and serving paradigms could be explored.

Summary updated

BibTeX
@inproceedings{zhang2025blitzscale,
  author = {Dingyan Zhang and Haotian Wang and Yang Liu and Xingda Wei and Yizhou Shan and Rong Chen and Haibo Chen},
  title = {{BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching}},
  booktitle = {OSDI},
  year = {2025},
}

Loading related content...

Figure