Practice of Alibaba cloud on elastic resource provisioning for large‐scale microservices cluster

Minxian Xu, Lei Yang, Yang Wang, Chengxi Gao, Linfeng Wen, Guoyao Xu, Liping Zhang, Kejiang Ye, Chengzhong Xu

Abstract

Managing resource provisioning for large-scale microservice architectures presents unique challenges, particularly when operating at a scale of over 10 million cores. Traditional reactive autoscaling often fails to account for complex service dependencies and the non-linear relationship between traffic and resource consumption. This paper details the implementation of Ali-Pro, an enhanced elastic resource provisioning framework deployed at Alibaba Cloud. By integrating LightGBM-based workload prediction with a Directed Acyclic Graph (DAG) optimization approach, Ali-Pro improves resource utilization by 10–15% while maintaining strict Service Level Objectives (SLOs) during peak traffic events like the Double 11 Shopping Festival.

Introduction

In modern cloud environments, scaling a handful of containers is a solved problem. However, managing 100,000 pods across 10,000 distinct applications introduces a level of complexity that standard tools, such as the Kubernetes Horizontal Pod Autoscaler (HPA), are not designed to handle. At Alibaba's scale, resource scheduling must navigate three primary obstacles:

  1. Deep Dependency Chains: Microservices often form calling chains exceeding 20 nodes. Under-provisioning a single intermediate service can cause cascading latency across the entire business logic.
  2. Resource Disparity: CPU and memory usage are frequently decoupled. Data from Alibaba’s production clusters shows that while 95% of services utilize less than 40% of their allocated CPU, nearly half of those same services require over 80% of their allocated memory.
  3. Operational Volatility: Global shopping events create sudden, massive bursts of traffic. Reactive scaling is often too slow, leading to service degradation before the infrastructure can catch up.

The Ali-Pro Framework

To address these challenges, Alibaba Cloud developed Ali-Pro, a hybrid provisioning system built on the Alibaba System Infrastructure (ASI). The framework utilizes a proactive-reactive strategy to balance efficiency with system stability.

Proactive Workload Prediction

Unlike traditional approaches that rely on ARIMA or LSTM models, Ali-Pro utilizes LightGBM, a gradient boosting framework. LightGBM proved more efficient at processing the high-dimensional, non-linear data characteristic of live production environments, providing more accurate forecasts of incoming traffic patterns.

Capacity and Optimization Modeling

To translate predicted traffic into hardware requirements, Ali-Pro employs two key mathematical models:

  • QPS Modeling: Using the Three-Sigma Rule ($\mu + 3\sigma$), the system models the capacity of individual pods to ensure they can handle 99.7% of expected Query Per Second (QPS) fluctuations.
  • Shortest Path Optimization: The provisioning problem is modeled as a Directed Acyclic Graph (DAG). Ali-Pro applies a shortest-path algorithm to determine the minimum number of replicates required to satisfy predicted demand across various time intervals, minimizing unnecessary resource churn.

Reactive Safety Mechanisms

Recognizing that no prediction is perfect, Ali-Pro includes a reactive scaling layer. This layer utilizes safety buffers and a defined "cooling time" to prevent "thrashing"—the rapid, inefficient cycling of adding and removing containers in response to transient noise.

Evaluation and Real-World Insights

The framework was validated using 15 TB of raw runtime data from Alibaba’s production clusters, specifically focusing on four representative E-business applications. The testing environment spanned over 10,000 nodes powered by Intel Xeon Platinum processors.

Key Findings

  • The Latency-QPS Paradox: The researchers observed that response times do not always increase with traffic. In many cases, higher QPS actually decreases latency because the increased load "warms up" Just-In-Time (JIT) compilers and caches. This finding suggests that simple threshold-based scaling can be counter-productive.
  • Memory as the Primary Constraint: While industry focus is often on CPU saturation, Alibaba’s data indicates that Out-of-Memory (OOM) errors are a more significant threat to microservice stability than CPU bottlenecks.
  • Efficiency Gains: Ali-Pro achieved a "Saved Percentage" (SP) of 18.1% compared to traditional static over-provisioning, nearing the theoretical optimal limit of 24%. This resulted in a total resource utilization increase of 10–15% without impacting the latency distribution.

Conclusion and Future Directions

Ali-Pro demonstrates that proactive, model-driven provisioning is essential for managing microservices at extreme scale. However, the authors identify several avenues for future research, including the management of "noisy neighbors" in co-located workloads and the expansion of the model to support heterogeneous hardware, such as GPUs and AI accelerators. Furthermore, deeper integration of cascading failure analysis into the scaling logic remains a priority for ensuring the resilience of increasingly complex dependency chains.

Summary updated

Abstract

Summary Cloud‐native architecture is becoming increasingly crucial for today's cloud computing environments due to the need for speed and flexibility in developing applications. It utilizes microservice technology to break down traditional monolithic applications into light‐weight and self‐contained microservice components. However, as microservices grow in scale and have dynamic inter‐dependencies, they also pose new challenges in resource provisioning that cannot be fully addressed by traditional resource scheduling approaches. The various microservices with different resource demands and latency requirements can create complex calling chains, making it difficult to provide fine‐grained and accurate resource allocation to each component while maintaining the overall quality of service in the chain. Alibaba Cloud has fully embraced cloud‐native and microservice technologies to drive its key business and scenarios, including Double 11 Shopping Festival. In this work, we aim to address the research problem on how to efficiently provision resources for the growing scale of microservice platform and ensure the performance of latency‐critical microservices. To address the problem, we present in‐depth analyses of Alibaba's microservice cluster and propose optimized resource provisioning algorithms to enhance resource utilization while ensuring the latency requirement. First, we analyze the distinct features of microservices in Alibaba's cluster compared to traditional applications. Then we present Alibaba's resource capacity provisioning workflow and framework to address challenges in resource provisioning for large‐scale and latency‐critical microservice clusters. Finally, we propose enhanced resource provisioning algorithms over Alibaba's current practice by making both proactive and reactive scheduling decisions based on different workloads patterns, which can improve resource usage by 10%–15% in Alibaba's clusters, while maintaining the necessary latency for microservices.
BibTeX
@inproceedings{xu2023practice,
  author = {Minxian Xu and Lei Yang and Yang Wang and Chengxi Gao and Linfeng Wen and Guoyao Xu and Liping Zhang and Kejiang Ye and Chengzhong Xu},
  title = {{Practice of Alibaba cloud on elastic resource provisioning for large‐scale microservices cluster}},
  booktitle = {Software},
  year = {2023},
  doi = {10.1002/spe.3271},
}
Figure