Anomaly Analysis for Co-located Datacenter Workloads in the Alibaba Cluster

Rui Ren, Jinheng Li, Lei Wang, Jianfeng Zhan, Zheng CaoarXiv 2018 · arXiv

Introduction: The IRU-QoS Trade-off

Cloud providers face a persistent challenge known as the "IRU-QoS curse." To maximize Infrastructure Resource Utilization (IRU) and reduce costs, providers co-locate latency-sensitive "online" applications (e.g., web servers) with "offline" batch processing jobs (e.g., data analytics) on the same physical hardware. While this strategy improves efficiency, it introduces performance interference and unpredictable behavior. This paper analyzes the 2017 Alibaba cluster trace to diagnose performance variance across identical hardware and identify the root causes of system anomalies.

Methodology: A Multi-Stage Analytical Pipeline

The researchers developed a robust pipeline to process and analyze 12 hours of production data from approximately 1,300 machines.

The multi-stage analytical pipeline for processing and analyzing cluster traces.
  1. Data Preprocessing: Missing resource data was addressed using linear interpolation. "Impossible" records—such as containers requesting excessive memory relative to the node's capacity—were filtered to ensure data integrity.
  2. Node Similarity Analysis: Dynamic Time Warping (DTW) was employed to compare machine behaviors. Unlike standard distance metrics, DTW accounts for temporal shifts, allowing the identification of nodes with similar usage patterns regardless of when specific tasks started.
  3. Workload Classification: Using K-means clustering, the machines were categorized into eight distinct workload types based on their specific mix of online and offline tasks.
  4. Anomaly Detection: The Isolation Forest (iForest) algorithm was used to detect outliers. By randomly partitioning high-dimensional data, iForest efficiently isolates anomalies, as outliers typically require fewer partitions to be separated from the main dataset.

Key Insights: Marathons vs. Sprints

The analysis revealed fundamental differences in how different workloads consume resources. Online containers behave like "marathon runners"—they are long-lived, stable, and primarily demand high memory. In contrast, batch jobs are "sprinters"—short-lived, CPU-intensive, and the source of significant, jagged fluctuations in resource utilization.

Resource utilization patterns across different machines, highlighting the volatility of CPU usage compared to memory.
  • Memory as the Primary Driver: While CPU scheduling often receives the most attention, the study found that memory demand from online containers is the dominant factor in defining a machine's workload "character."
  • The "Type 2" Scheduling Mystery: A cluster of machines (Type 2) was identified as being perfectly healthy yet hosting zero workloads. This suggests that inefficiencies in the global scheduler, rather than hardware failures, are a significant source of underutilization.
  • Anomalies Beyond Hardware Failure: The most frequent causes of anomalies were not broken components but "soft-errors" (transient system glitches) and "unbalanced scheduling," which inadvertently creates performance "hot spots" or "cold spots" across the cluster.

Results and Diagnostic Mapping

The study successfully classified 72.8% of nodes as "Type 1," representing successful co-location of online and offline jobs. The iForest algorithm flagged 81 machines with significant anomaly scores. By mapping these anomalies to specific events, the researchers identified that many "active" nodes were failing to run jobs due to frequent soft-errors. This "performance tilt"—where resource distribution becomes skewed—is an inevitable consequence of dense multi-application deployment.

Conclusion and Future Directions

This research provides a critical diagnostic framework for understanding the complexities of modern cloud environments. While the 12-hour Alibaba trace offered groundbreaking visibility, future work requires longer temporal datasets to capture cyclical patterns. Furthermore, while this paper identifies scheduling as a primary source of anomalies, the next step for the industry is developing adaptive scheduling algorithms that can proactively mitigate these imbalances to break the IRU-QoS curse.

Summary updated

Abstract

In warehouse-scale cloud datacenters, co-locating online services and offline batch jobs is an efficient approach to improving datacenter utilization. To better facilitate the understanding of interactions among the co-located workloads and their real-world operational demands, Alibaba recently released a cluster usage and co-located workload dataset, which is the first publicly dataset with precise information about the category of each job. In this paper, we perform a deep analysis on the released Alibaba workload dataset, from the perspective of anomaly analysis and diagnosis. Through data preprocessing, node similarity analysis based on Dynamic Time Warping (DTW), co-located workloads characteristics analysis and anomaly analysis based on iForest, we reveals several insights including: (1) The performance discrepancy of machines in Alibaba's production cluster is relatively large, for the distribution and resource utilization of co-located workloads is not balanced. For instance, the resource utilization (especially memory utilization) of batch jobs is fluctuating and not as stable as that of online containers, and the reason is that online containers are long-running jobs with more memory-demanding and most batch jobs are short jobs, (2) Based on the distribution of co-located workload instance numbers, the machines can be classified into 8 workload distribution categories1. And most patterns of machine resource utilization curves are similar in the same workload distribution category. (3) In addition to the system failures, unreasonable scheduling and workload imbalance are the main causes of anomalies in Alibaba's cluster.
BibTeX
@misc{ren2018anomaly,
  author = {Rui Ren and Jinheng Li and Lei Wang and Jianfeng Zhan and Zheng Cao},
  title = {{Anomaly Analysis for Co-located Datacenter Workloads in the Alibaba Cluster}},
  year = {2018},
  eprint = {1811.06901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.06901},
}
Figure