Abstract
Microservices have transformed monolithic applications into lightweight, self-contained, and isolated application components, establishing themselves as a dominant paradigm for application development and deployment in public clouds such as Google and Alibaba. Autoscaling emerges as an efficient strategy for managing resources allocated to microservices' replicas. However, the dynamic and intricate dependencies within microservice chains present challenges to the effective management of scaled microservices. Additionally, the centralized autoscaling approach can encounter scalability issues, especially in the management of large-scale microservice-based clusters. To address these challenges and enhance scalability, we propose an innovative distributed resource provisioning approach for microservices based on the Twin Delayed Deep Deterministic Policy Gradient algorithm. This approach enables effective autoscaling decisions and decentralizes responsibilities from a central node to distributed nodes. Comparative results with state-of-the-art approaches, obtained from a realistic testbed and traces, indicate that our approach reduces the average response time by 15% and the number of failed requests by 24%, validating improved scalability as the number of requests increases.
Introduction: The Bottleneck of Centralized Scaling
Microservices have become the standard for building scalable cloud applications. However, because microservices are highly interdependent, determining exactly when and how to scale them—whether by adding more instances (horizontal scaling) or allocating more CPU/memory (vertical scaling)—is a complex optimization problem.
Recently, Reinforcement Learning (RL) has been used to automate these scaling decisions. However, most existing RL solutions are centralized: a single central node collects all cluster data, runs heavy RL inference, and dictates scaling actions. When traffic spikes, this central node becomes a massive bottleneck, suffering from latency and unreliability that leads to delayed scaling and degraded user experience. As shown in the comparison below, most existing methods lack a distributed decision-making pattern and fail to integrate multi-dimensional scaling techniques like brownout.
The DRPC Framework
To address these limitations, we propose DRPC (Distributed Reinforcement Learning Approach for Scalable Resource Provisioning in Container-based Clusters). DRPC shifts from a centralized command center to a distributed, asynchronous architecture using a "Teacher-Student" model.
System Architecture
The framework consists of three main components designed to decentralize the decision-making process:
- Workload Predictor: Uses a Gated Recurrent Unit (GRU) neural network to forecast incoming traffic based on historical data, allowing the system to act proactively.
- Central Teacher Network: A centralized RL agent powered by the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm. It explores the global state of the cluster and learns optimal policies for continuous, high-dimensional scaling actions.
- Distributed Student Networks: Lightweight neural networks deployed alongside individual microservices. Using Imitation Learning, these "Students" learn to mimic the Teacher's optimal policies. Once trained, they make rapid, asynchronous, local scaling decisions without consulting the central node.
A Retraining Notifier monitors local rewards and triggers the Teacher to retrain if the Students' performance drops due to shifting workload patterns, ensuring the system remains robust over time.
Experimental Setup and Results
We evaluated DRPC using real-world Alibaba Cluster Traces (4,000 machines over 8 days) and the TrainTicket benchmark, an open-source application with ~40 interconnected services deployed on a Kubernetes cluster.
Key Performance Metrics
Tested against state-of-the-art centralized RL autoscalers (FIRM, CoScal) and standard Kubernetes scaling (KuScal), DRPC demonstrated superior performance:
- Response Times: Reduced average response time by 15% compared to FIRM and 24% compared to CoScal. DRPC consistently maintained response times between 40ms and 70ms.
- Reliability: Slashed the request failure rate by 24% vs. FIRM, 44% vs. CoScal, and 85% vs. KuScal.
- Computational Efficiency: DRPC requires significantly less overhead, using just $6 \times 10^{-5}$ units of CPU time for decision-making, compared to $4 \times 10^{-3}$ for FIRM.
- Quality of Service (QoS): Maintained high QoS, ensuring 99% of requests were handled within 210 ms.
Conclusion
DRPC demonstrates that decentralizing inference to lightweight "Student" networks allows cloud systems to react to load changes asynchronously and in parallel. By combining global optimization via a Teacher network with local execution speed via Student networks, DRPC bypasses the central node latency that often cripples large-scale microservice clusters during traffic spikes. Future work will focus on automating timing intervals for updates and exploring anomaly-aware workload management in even larger production environments.
Summary updated