Abstract
Co-locating latency-critical services (LCSs) and best-effort jobs (BEJs) constitute the principal approach for enhancing resource utilization in production. Nevertheless, the co-location practice hurts the performance of LCSs due to resource competition, even when employing isolation technology. Through an extensive analysis of voluminous real trace data derived from two production clusters, we observe that BEJs typically exhibit periodic execution patterns and serve as the primary sources of interference to LCSs. Furthermore, despite occupying the same level of resource consumption, the diverse compositions of BEJs can result in varying degrees of interference on LCSs. Subsequently, we propose PISM, a proactive Performance Interference Scoring and Mitigating framework for LCSs through the optimization of BEJ scheduling. Firstly, PISM adopts a data-driven approach to establish a characterization and classification methodology for BEJs. Secondly, PISM models the relationship between the composition of BEJs on servers and the response time (RT) of LCSs. Thirdly, PISM establishes an interference scoring mechanism in terms of RT, which serves as the foundation for BEJ scheduling. We assess the effectiveness of PISM on a small-scale cluster and through extensive data-driven simulations. The experiment results demonstrate that PISM can reduce cluster interference by up to 41.5%, and improve the throughput of long-tail LCSs by 76.4%.
Introduction
In large-scale cloud environments, efficiency is a balancing act. To keep costs down, providers co-locate two very different types of workloads on the same servers: Latency-Critical Services (LCSs) and Best-Effort Jobs (BEJs). However, this practice often leads to performance degradation for LCSs due to competition for shared, unmanaged resources such as L3 cache and memory bandwidth. Even with container-based isolation, "noisy neighbor" effects persist, causing unpredictable latency spikes in critical user-facing applications.
As shown in Figure 2, the presence of BEJs significantly shifts the response time distribution of LCSs, increasing tail latency. Existing mitigation strategies are often reactive or rely on static benchmarks that fail to capture the dynamic nature of production clusters. There is a critical need for a proactive, data-driven approach to predict and mitigate this interference.
The PISM Framework
We propose the Performance Interference Scoring and Mitigating (PISM) framework. PISM leverages the repetitive nature of production workloads to proactively schedule BEJs in a way that minimizes impact on LCSs. The framework consists of three core components:
- BEJ Characterization and Classification: PISM categorizes BEJs into 60 distinct types based on their CPU usage, memory footprint, and duration. By utilizing Graph Kernels to analyze Directed Acyclic Graph (DAG) structures, PISM identifies recurring job patterns to predict their resource behavior.
- Interference Modeling and Scoring: Rather than relying solely on raw utilization metrics, PISM models the specific relationship between the composition of BEJs on a server and the resulting RT of co-located LCSs. This relationship is quantified into an interference score (1–10).
- Interference-Aware Scheduling: The scheduler uses these scores to evaluate candidate servers for new BEJs, selecting placements that minimize the predicted interference score for existing microservices.
Figure 6 illustrates the positive correlation between BEJ CPU utilization and LCS response time, providing the empirical basis for the PISM scoring mechanism.
Evaluation and Results
PISM was evaluated using massive production traces from Alibaba and Google, as well as a small-scale Kubernetes testbed. Key findings include:
- Interference Reduction: PISM reduced overall cluster interference by up to 41.5% compared to standard load-balancing schedulers.
- Throughput Improvement: For long-tail services, throughput improved by 76.4%.
- Latency Mitigation: In the Kubernetes testbed, P99 latency for a "checkout" service was reduced from 8000ms to 3000ms.
- Prediction Accuracy: The framework achieved 87.2% accuracy in predicting interference scores for critical services.
Conclusion
PISM demonstrates that a data-driven, proactive scoring mechanism can effectively manage microservice interference in co-located environments. By focusing on the composition of workloads rather than just aggregate resource quantity, PISM enables higher utilization without sacrificing the performance of latency-critical applications. Future work will explore LCS-to-LCS interference and integrate network and I/O metrics into the scoring model.
Summary updated