TL;DR
Modern datacenters pack many jobs onto the same machines, but some jobs can hog resources and slow others down (antagonists). This paper introduces PANDA, a system that accurately identifies these antagonistic jobs in production environments by using global historical data and a new machine-level performance metric, overcoming the noise and multi-victim challenges of previous methods.
What Problem Does This Paper Address?
Warehouse-scale datacenters aim for high resource utilization by running multiple jobs on the same physical machines. However, this can lead to "performance interference," where some "antagonistic" jobs consume excessive shared resources, negatively impacting the performance of other co-located jobs, especially latency-sensitive ones. Existing methods for detecting these antagonists fall into two camps:
- Offline Profiling: This involves setting up dedicated test environments to profile each job individually. While accurate, it's incredibly costly and doesn't scale to millions of jobs, and job behaviors can change over time, requiring constant re-profiling.
- Sample-from-Production: This approach collects performance data directly from live machines. It's more scalable but suffers from significant "sampling noise" due to short measurement windows needed to avoid impacting production workloads. This noise can lead to false alarms and, critically, misidentification of antagonists. Furthermore, modern datacenters often have many jobs per machine, leading to "multi-victim scenarios" where multiple jobs are affected simultaneously, making it hard to pinpoint the true antagonist when analyzing victims individually.
This paper addresses the limitations of these existing methods, aiming for a scalable, noise-resilient, and accurate antagonist identification system for production datacenters.
Methods
PANDA introduces a novel framework with two core components: an offline phase for data collection and analysis, and an online phase for real-time identification.
- Global View & Job-Level Inference: Instead of focusing on individual tasks or local machine history, PANDA aggregates information globally across all machines and over extended time periods. It also shifts the focus from identifying antagonistic tasks to identifying antagonistic jobs, recognizing that tasks within the same job share similar characteristics.
- Machine-Level Performance Metric: PANDA uses a new metric called
mnCPI(machine-level normalized CPI). This metric represents the average normalized CPI of all latency-sensitive (LS) jobs running on a given machine. By aggregating at the machine level, PANDA smooths out noise from individual task anomalies and effectively captures shared resource contention that impacts multiple jobs. This also helps address the multi-victim problem by providing a holistic view of machine performance degradation. - Antagonist Score Calculation: PANDA calculates an "antagonist score" for each job. This score is derived from the correlation between a job's CPU utilization and the
mnCPIof the machines it co-locates on. This correlation is weighted by the job's CPU usage, emphasizing periods where the job is actively consuming resources and thus more likely to cause interference. - Noise Suppression: The global aggregation and machine-level metric are key to suppressing sampling noise. By looking at broader historical trends and machine-wide effects, PANDA can distinguish genuine interference from transient measurement fluctuations.
Datasets
The evaluation uses a large-scale production trace from Google (Google, 2019), specifically "cell a." This dataset contains approximately 1 million tasks executed across 80,000 production machines over 31 days, providing detailed per-task resource usage and performance statistics. No new benchmarks were introduced; the evaluation leverages existing production data.
Key Insights and Ideas
- Global Context is Crucial: The paper strongly demonstrates that looking beyond a single machine's recent history is vital. Aggregating data globally across all machines and time significantly improves the accuracy and stability of antagonist identification by averaging out local noise.
- Job-Level Focus Simplifies and Strengthens: Identifying antagonistic jobs rather than individual tasks is a more robust approach. Since tasks within a job are similar, their collective behavior is a better indicator of interference, and it simplifies the problem space.
- Machine-Level Metrics as Noise Filters: Using a machine-level performance metric like
mnCPIeffectively acts as a noise filter. It smooths out individual task anomalies and highlights genuine interference that affects the entire machine, which is often caused by severe resource contention. - Multi-Victim Scenarios are Common and Challenging: The research highlights that in modern datacenters, multiple victims often appear simultaneously. Analyzing each victim independently leads to inconsistent and unreliable antagonist identification, underscoring the need for a holistic approach like PANDA's.
Key Results
PANDA significantly outperforms prior correlation-based methods in identifying true antagonists:
- Improved Ranking: PANDA ranks true antagonists at an average suspicion percentile of 82.6%, a substantial improvement over prior methods which achieved only 50-55%. This means PANDA is much more likely to place the actual antagonist at the top of its ranked list of potential culprits.
- Consistent Identification in Multi-Victim Scenarios: PANDA achieves a 100% consistency rate in multi-victim scenarios, meaning it consistently identifies the same antagonist even when multiple jobs are affected. In contrast, baseline methods showed significant disagreement (e.g., 31.4% and 37.6% consistency for CPI² and Proctor, respectively).
- Negligible Runtime Overhead: PANDA's design allows for efficient online monitoring with minimal runtime overhead, making it practical for production deployment.
Limitations and Future Work
- Ground Truth Limitations: The paper acknowledges that establishing perfect ground truth in production environments is challenging. Their evaluation relies on human expert labeling, which is inherently one-sided (experts can confirm antagonists but not definitively rule them out). This limits the use of standard classification metrics like recall and focuses the evaluation on precision and ranking.
- Future Work: While not explicitly detailed, the success of PANDA in improving accuracy and robustness suggests future work could explore integrating PANDA's insights into proactive scheduling decisions to prevent interference before it occurs, or further refining the machine-level metric to capture more nuanced resource contention. The paper's focus on noise resilience and multi-victim handling lays a strong foundation for such advancements.
Summary updated