Mitigating interference of microservices with a scoring mechanism in large-scale clusters

Dingyu Yang, Kangpeng Zheng, Shiyou Qian, Qin Hua, Kaixuan Zhang, Jian Cao, Guangtao Xue

In the world of large-scale cloud computing, efficiency is a balancing act. To keep costs down, providers co-locate two very different types of workloads on the same servers: Latency-Critical Services (LCSs)—like the "checkout" button on an e-commerce site that needs to be instant—and Best-Effort Jobs (BEJs)—like background data processing that can take its time. The problem? These roommates don't always get along. When a heavy background job hogs the CPU cache, your "instant" checkout button suddenly takes three seconds to respond.

TL;DR

This paper introduces PISM, a proactive framework that uses a data-driven scoring mechanism to predict and mitigate performance interference in large-scale clusters. By classifying background jobs based on their historical resource "fingerprints," PISM allows schedulers to tuck them into servers where they’ll do the least harm, improving long-tail service throughput by a staggering 76.4%.

What Problem Does This Paper Address?

The industry standard for improving resource utilization is co-location. However, even with modern isolation technologies (like containers), applications still compete for shared "unmanaged" resources like L3 cache and memory bandwidth.

Existing solutions often fall into two traps: they are either reactive (waiting for a latency spike to happen before moving a job) or limited (relying on pre-run benchmarks that don't reflect the messy reality of a production cluster). In a dynamic environment like Alibaba's or Google's, where thousands of new tasks are submitted every minute, we need a way to predict interference before it happens without the overhead of constant, invasive monitoring.

Methods

The authors propose the Performance Interference Scoring and Mitigating (PISM) framework, which operates in three main stages:

  1. BEJ Characterization & Classification: Instead of treating all background jobs as a monolith, PISM categorizes them into 60 distinct types based on a "triple" of features: CPU usage, Memory usage, and Makespan (duration). It uses Graph Kernels to identify similar jobs based on their Directed Acyclic Graph (DAG) structures, allowing the system to "remember" how a specific type of job behaved in the past.
  2. Interference Scoring: PISM translates complex performance metrics into a simple 1–10 score. It maps the Response Time (RT) of LCSs to these levels. A score of 1 means "smooth sailing," while 10 indicates a high risk of resource contention.
  3. Interference-Aware Scheduling: When a new background job arrives, the scheduler doesn't just look for a server with free CPU cycles. It queries the Interference Scorer to predict the impact on the specific mix of microservices already running on candidate servers and picks the path of least resistance.

Datasets

The study is grounded in massive, real-world scale:

  • Alibaba Production Trace: Data from 10,000 servers running ~25,000 microservices and ~4 million background jobs per day.
  • Google 2019 Cluster Data: A public dataset from a 12,000-server cluster used to validate the generalizability of their observations.
  • Small-scale Testbed: A 14-server Kubernetes cluster using "Online Boutique" (a microservice demo) and Apache Spark to verify the framework in a controlled environment.

Key Insights and Ideas

  • Composition Over Quantity: One of the most "aha" moments in the paper is that total CPU utilization is a poor predictor of interference. Two servers might both be at 60% CPU usage, but if one is running memory-heavy "Join" tasks and the other is running CPU-bound "Map" tasks, the impact on a co-located microservice will be vastly different.
  • The Repeatability Goldmine: The authors observed that background jobs in production are highly repetitive. Most belong to recurring DAG structures (like daily cron jobs). This repeatability makes it possible to use historical data to predict the future behavior of "new" tasks.
  • Non-Invasive is Key: PISM doesn't require deploying heavy agents on every server. By using high-level metadata and historical traces, it remains lightweight enough for production use.

Key Results

The experimental results show that PISM significantly outperforms standard scheduling logic:

  • Interference Reduction: PISM reduced overall cluster interference by up to 41.5% compared to baseline schedulers.
  • Tail Latency Mastery: For "tail-heavy" services (those prone to occasional massive slowdowns), PISM improved throughput by 76.4%.
  • Prediction Accuracy: The framework achieved an 87.2% accuracy in predicting interference scores for the critical "checkout" service.
  • Long-Tail Mitigation: In the Kubernetes testbed, the P99 latency (the worst 1% of requests) for the checkout service was slashed from 8000ms to 3000ms.

Limitations and Future Work

While PISM is powerful, the authors note a few areas for growth:

  • Boundary Errors: The classification system can struggle with tasks that sit right on the edge of resource buckets (e.g., a task using exactly 0.5 cores).
  • LCS-to-LCS Interference: Currently, PISM focuses on how background jobs affect microservices. It doesn't yet account for how two different latency-critical microservices might interfere with each other.
  • External Factors: The model primarily considers compute resources; interference from network congestion or disk I/O is not yet integrated into the scoring mechanism.

For engineers managing "noisy neighbor" problems in the cloud, PISM offers a compelling blueprint: stop looking at just how much resource is being used, and start looking at who is using it.

Summary updated

Abstract

Co-locating latency-critical services (LCSs) and best-effort jobs (BEJs) constitute the principal approach for enhancing resource utilization in production clusters. Nevertheless, the co-location practice hurts the performance of LCSs due to resource competition, even when employing isolation technology. Through an extensive analysis of voluminous real trace data derived from two production clusters, we observe that BEJs typically exhibit periodic execution patterns and serve as the primary sources of interference to LCSs. Furthermore, despite occupying the same level of resource consumption, the diverse compositions of BEJs can result in varying degrees of interference on LCSs. Subsequently, we propose PISM, a proactive Performance Interference Scoring and Mitigating framework for LCSs through the optimization of BEJ scheduling. Firstly, PISM adopts a data-driven approach to establish a characterization and classification methodology for BEJs. Secondly, PISM models the relationship between the composition of BEJs on servers and the response time (RT) of LCSs. Thirdly, PISM establishes an interference scoring mechanism in terms of RT, which serves as the foundation for BEJ scheduling. We assess the effectiveness of PISM on a small-scale cluster and through extensive data-driven simulations. The experiment results demonstrate that PISM can reduce cluster interference by up to 41.5%, and improve the throughput of long-tail LCSs by 76.4%.
BibTeX
@article{yang2025mitigating,
  author = {Dingyu Yang and Kangpeng Zheng and Shiyou Qian and Qin Hua and Kaixuan Zhang and Jian Cao and Guangtao Xue},
  title = {{Mitigating interference of microservices with a scoring mechanism in large-scale clusters}},
  journal = {Journal of Supercomputing},
  year = {2025},
  doi = {10.1007/s11227-024-06534-7},
}

Loading related content...

Figure