Abstract
Current Large Language Model (LLM) serving systems typically scale by replicating entire monolithic model instances. This approach is often slow to react to traffic bursts and leads to significant resource waste due to the heterogeneous resource demands of internal model components. We present a novel operator-level autoscaling framework that treats models as graphs of individual operators. By selectively scaling only the bottleneck components, this approach reduces GPU requirements by up to 40% and energy consumption by 35% while maintaining strict Service-Level Objectives (SLOs).
Introduction: The Inefficiency of Monolithic Scaling
Serving LLMs requires balancing strict performance targets—such as Time to First Token (TTFT) and Time Between Tokens (TBT)—against the high costs of GPU infrastructure. Traditional "model-level" autoscaling is the industry standard but suffers from two major drawbacks:
- Coarse Granularity: Models consist of diverse operators (attention, normalization, linear layers) with varying resource needs. Scaling the entire model forces over-provisioning of lightweight operators to support a single bottleneck.
- High Latency: Loading large models (e.g., 70B parameters) can take upwards of 10 seconds, making it impossible to respond to rapid traffic fluctuations without dropping requests or violating SLOs.
Rethinking Granularity: Operator-Level Autoscaling
The proposed framework shifts the unit of scaling from the entire model to individual operators. This fine-grained control allows the system to target specific bottlenecks on the model's critical path.
The system employs a greedy autoscaling algorithm based on queue modeling to identify bottleneck operators and selectively adjust their replicas or parallelism. To manage the resulting complexity, a contention-aware placement strategy packs these independent "shards" onto GPUs, accounting for memory limits and performance interference.
Key Insights: Exploiting Operator Heterogeneity
A core finding of this research is that not all parts of a model scale equally. For instance, attention operators exhibit quadratic complexity during the prefill phase, whereas linear layers scale linearly.
By exploiting this heterogeneity, the system can allocate hardware more efficiently. The "prefill" stage, which is compute-intensive and bursty, sees 2–3x more benefit from fine-grained scaling than the "decode" stage. Furthermore, as models grow in size, the opportunities for co-scheduling diverse operators increase, mitigating GPU resource fragmentation.
Results and Impact
Evaluated against production traces from Microsoft Azure and Alibaba's Mooncake platform, the operator-level approach demonstrated significant improvements:
- Resource Efficiency: Achieved the same SLOs using 40% fewer GPUs.
- Sustainability: Reduced energy consumption by 35%.
- Throughput: Delivered 1.6x higher throughput under fixed resource constraints.
- Memory Savings: Saved up to 60% memory for long-context workloads by avoiding unnecessary replication of non-sensitive operators.
Conclusion and Future Directions
Operator-level autoscaling represents a significant step toward more efficient and responsive LLM serving. While highly effective, future work remains in co-optimizing this execution-layer scaling with request schedulers and addressing constraints imposed by "megakernel fusion" in some inference engines. Additionally, minimizing communication overhead between operators placed on different GPUs via topology-aware placement is a key area for further refinement.
Summary updated