Abstract
Resource usage of production workloads running on shared compute clusters often fluctuate significantly across time. While simultaneous spike in the resource usage between two workloads running on the same machine can create performance degradation, unused resources in a machine results in wastage and undesirable operational characteristics for a compute cluster. Prior works did not consider such temporal resource fluctuations or their alignment for scheduling decisions. Due to the variety of time-varying workloads, their complex resource usage characteristics, it is challenging to design well-defined heuristics for scheduling them optimally across different machines in a cluster. In this paper, we propose a Deep Reinforcement Learning (DRL) based approach to exploit various temporal resource usage patterns of time varying workloads as well as a technique for creating equivalence classes among a large number of production workloads to improve scalability of our method. Validations with real production traces from Google and Alibaba show that our technique can significantly improve metrics for operational excellence (e.g. utilization, fragmentation, resource exhaustion etc.) for a cluster, compared to the baselines.
Introduction
In massive data centers, such as those operated by Google or Alibaba, microservices are densely packed onto shared physical machines to optimize infrastructure costs. However, most of these services are Time-Varying Workloads (TVWs), characterized by resource demands (CPU and memory) that fluctuate significantly over time due to user behavior or internal algorithmic phases.
Traditional schedulers, like those found in Kubernetes or Mesos, often treat resource requests as static peak limits. This approach leads to two primary inefficiencies:
- Resource Contention: Simultaneous spikes from multiple workloads on the same machine can degrade performance.
- Resource Wastage: Over-provisioning to avoid contention results in high fragmentation and under-utilized hardware.
The complexity of these temporal patterns makes it difficult to design effective "if-then" heuristics. This research introduces TVW-RL, a system that learns to recognize the "rhythm" of workloads to optimize placement.
Methods: The TVW-RL Framework
The authors frame cluster scheduling as a sequential decision-making problem solved via Deep Reinforcement Learning (DRL).
Core Components
- State Representation: The agent observes a 2D matrix representing the resource usage history over the last $h$ timesteps. This allows the model to perceive the temporal "shapes" (spikes and dips) of active jobs.
- Multi-Objective Reward Function: The agent is trained to minimize contention (spiky jobs together), under-utilization, overshoot (exceeding capacity), and queue wait times.
- Scalability via Equivalence Classes (EC): To manage thousands of unique jobs, the system groups them into "Equivalence Classes" using Dynamic Time Warping (DTW). This reduces the complexity of the action space, as the agent learns to schedule behavior types rather than individual services.
Experimental Evaluation and Results
The system was evaluated using real-world production traces: the Google Cluster Traces (2011) and the Alibaba Cluster Traces (2018). The study focused on long-running jobs (over 30 hours) where long-term placement impact is most critical.
TVW-RL demonstrated significant improvements over state-of-the-art baselines like Tetris and DeepRM:
- Utilization: Average CPU and memory utilization increased by 30-300% compared to DeepRM and 68-100% over Tetris.
- Infrastructure Efficiency: The number of physical machines required was reduced by 8-50% depending on the load.
- Fragmentation and Robustness: Resource fragmentation was reduced by 5-13%, and the system maintained performance even under unpredictable background noise.
- Overshoot Prevention: Unlike greedy heuristics that often lead to resource exhaustion during spikes, TVW-RL learns to leave "breathing room," resulting in near-zero overshoot.
Discussion and Future Work
The primary insight of this work is that optimal scheduling depends on temporal alignment—matching a job that spikes at noon with one that spikes at midnight.
While highly effective, the approach has limitations. The offline training process is computationally intensive (28-108 hours), which may hinder rapid adaptation to novel workloads. Additionally, the current model focuses on initial placement; future iterations could incorporate "re-balancing" migrations to further enhance efficiency. The fixed history window ($h$) also presents a challenge for workloads with very long-term seasonal patterns (e.g., weekly cycles), suggesting a need for more flexible temporal windows in future research.
Summary updated