From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models

Xingqi Cui, Chieh-Jan Mike Liang, Jiarong Xing, Haoran QiuarXiv 2025 · DOI · arXiv

Abstract

Current Large Language Model (LLM) serving systems typically scale by replicating entire monolithic model instances. This approach is often slow to react to traffic bursts and leads to significant resource waste due to the heterogeneous resource demands of internal model components. We present a novel operator-level autoscaling framework that treats models as graphs of individual operators. By selectively scaling only the bottleneck components, this approach reduces GPU requirements by up to 40% and energy consumption by 35% while maintaining strict Service-Level Objectives (SLOs).

Introduction: The Inefficiency of Monolithic Scaling

Serving LLMs requires balancing strict performance targets—such as Time to First Token (TTFT) and Time Between Tokens (TBT)—against the high costs of GPU infrastructure. Traditional "model-level" autoscaling is the industry standard but suffers from two major drawbacks:

  1. Coarse Granularity: Models consist of diverse operators (attention, normalization, linear layers) with varying resource needs. Scaling the entire model forces over-provisioning of lightweight operators to support a single bottleneck.
  2. High Latency: Loading large models (e.g., 70B parameters) can take upwards of 10 seconds, making it impossible to respond to rapid traffic fluctuations without dropping requests or violating SLOs.

Rethinking Granularity: Operator-Level Autoscaling

The proposed framework shifts the unit of scaling from the entire model to individual operators. This fine-grained control allows the system to target specific bottlenecks on the model's critical path.

Comparison of model-level vs. operator-level autoscaling. Model-level scaling replicates the entire graph, while operator-level scaling targets specific bottlenecks.

The system employs a greedy autoscaling algorithm based on queue modeling to identify bottleneck operators and selectively adjust their replicas or parallelism. To manage the resulting complexity, a contention-aware placement strategy packs these independent "shards" onto GPUs, accounting for memory limits and performance interference.

Key Insights: Exploiting Operator Heterogeneity

A core finding of this research is that not all parts of a model scale equally. For instance, attention operators exhibit quadratic complexity during the prefill phase, whereas linear layers scale linearly.

Normalized latency across different operators for various LLM architectures. The 'attn' operator often shows significantly different scaling behavior compared to others.

By exploiting this heterogeneity, the system can allocate hardware more efficiently. The "prefill" stage, which is compute-intensive and bursty, sees 2–3x more benefit from fine-grained scaling than the "decode" stage. Furthermore, as models grow in size, the opportunities for co-scheduling diverse operators increase, mitigating GPU resource fragmentation.

Results and Impact

Evaluated against production traces from Microsoft Azure and Alibaba's Mooncake platform, the operator-level approach demonstrated significant improvements:

  • Resource Efficiency: Achieved the same SLOs using 40% fewer GPUs.
  • Sustainability: Reduced energy consumption by 35%.
  • Throughput: Delivered 1.6x higher throughput under fixed resource constraints.
  • Memory Savings: Saved up to 60% memory for long-context workloads by avoiding unnecessary replication of non-sensitive operators.

Conclusion and Future Directions

Operator-level autoscaling represents a significant step toward more efficient and responsive LLM serving. While highly effective, future work remains in co-optimizing this execution-layer scaling with request schedulers and addressing constraints imposed by "megakernel fusion" in some inference engines. Additionally, minimizing communication overhead between operators placed on different GPUs via topology-aware placement is a key area for further refinement.

Summary updated

Abstract

Serving large generative models such as LLMs and multi- modal transformers requires balancing user-facing SLOs (e.g., time-to-first-token, time-between-tokens) with provider goals of efficiency and cost reduction. Existing solutions rely on static provisioning or model-level autoscaling, both of which treat the model as a monolith. This coarse-grained resource management leads to degraded performance or significant resource underutilization due to poor adaptability to dynamic inference traffic that is common online. The root cause of this inefficiency lies in the internal structure of generative models: they are executed as graphs of interconnected operators. Through detailed characterization and systematic analysis, we find that operators are heterogeneous in their compute and memory footprints and exhibit diverse sensitivity to workload and resource factors such as batch size, sequence length, and traffic rate. This heterogeneity suggests that the operator, rather than the entire model, is the right granularity for scaling decisions. We propose an operator-level autoscaling framework, which allocates resources at finer (operator)-granularity, optimizing the scaling, batching, and placement based on individual operator profiles. Evaluated on production-scale traces, our approach preserves SLOs with up to 40% fewer GPUs and 35% less energy, or under fixed resources achieves 1.6x higher throughput with 5% less energy. These results show that the operator, rather than the model, is fundamentally a more effective unit for scaling large generative workloads.
BibTeX
@misc{cui2025from,
  author = {Xingqi Cui and Chieh-Jan Mike Liang and Jiarong Xing and Haoran Qiu},
  title = {{From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models}},
  year = {2025},
  eprint = {2511.02248},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.02248},
}
Figure