CCA: An ML Pipeline for Cloud Anomaly Troubleshooting

Lili Georgieva, Ioana Giurgiu, Serge Monney, Haris Pozidis, Viviane Potocnik, Mitch GusatAAAI 2022 · DOI

Abstract

The Cloud Causality Analyzer (CCA) is an automated machine learning pipeline designed for Root Cause Analysis (RCA) in massive cloud storage environments. By integrating smart dimensionality reduction with a hierarchical causal inference ensemble, CCA identifies performance bottlenecks using 15x fewer metrics than traditional methods while maintaining high diagnostic accuracy. This paper details the three-stage architecture that bridges the gap between symptom detection and interpretable root cause identification.

Introduction

Modern cloud infrastructures generate thousands of Key Performance Indicators (KPIs) every second. When performance anomalies occur—such as sudden latency spikes—Site Reliability Engineers (SREs) are often overwhelmed by a "data tsunami." The challenge is twofold: processing thousands of multivariate time-series in real-time is computationally prohibitive, and identifying the specific component that triggered a chain reaction requires deep explainability. CCA addresses these challenges by automating the path from raw data to a causal graph of the failure.

Methodology

The CCA system is structured as a three-stage pipeline designed to filter noise and extract causal relationships.

The three-stage CCA pipeline architecture, spanning from front-end dimensionality reduction to back-end root cause analysis.

1. Front-end (DimRedux)

Instead of monitoring every KPI, CCA employs $k$-shape clustering to group similar metrics. It selects "Representative KPIs" (R-KPIs) by picking both "central" metrics (representing normal behavior) and "peripheral" metrics. Selecting peripheral KPIs is a key innovation, as these metrics are the most likely to exhibit the abnormalities necessary for effective detection.

2. Mid-end (Embedded AD)

The selected R-KPIs are processed by a Temporal Convolutional Network (TCN)-based Autoencoder. TCNs are utilized for their ability to handle long-range temporal dependencies. The model learns to reconstruct normal patterns; when the reconstruction residual exceeds a threshold, an anomaly is flagged.

3. Back-end (Explainability & RCA)

This stage identifies the "culprits" behind the anomaly using a hierarchical ensemble of explainability tools. It moves beyond simple correlation to establish a "causal chain" that engineers can follow to the source of the failure.

Causal Inference and Explainability

The back-end uses a combination of techniques to distinguish between root causes and downstream symptoms:

  • SHAP-XAD: Utilizes SHAP values to identify which features contributed most to the anomaly detection score.
  • Granger-PVG & cTCDF: These models build causal graphs to determine the direction of influence (e.g., determining if a change in "Back-end Write I/O Rate" caused a spike in "Response Time").
An example of the CCA Ensemble Causal Graph, showing the troubleshooting window and the identified causal links between performance metrics.

Results and Operational Impact

The system was evaluated using real-world data from IBM’s Cloud storage environments, covering over 10,000 large-scale systems. Key findings include:

  • Efficiency: CCA achieved its results using 15x fewer KPIs than the reference baseline, significantly reducing computational overhead.
  • Accuracy: The system reached an average accuracy of ~67% (peaking at 88% in specific scenarios) when compared against ground-truth labels from human Subject Matter Experts (SMEs).
  • Search Space Reduction: The pipeline successfully narrowed down hundreds of potential metrics to a handful of explanatory KPIs, drastically reducing the time required for manual troubleshooting.

Conclusion

CCA demonstrates that combining dimensionality reduction with causal ensembling can effectively manage the scale of modern cloud monitoring. While accuracy varies by scenario complexity, the 15x reduction in data volume and the generation of interpretable causal graphs represent a significant step toward automated, self-healing cloud infrastructures. Future work will focus on refining clustering logic and integrating the pipeline with automated remediation systems.

Summary updated

BibTeX
@inproceedings{georgieva2022cca,
  author = {Lili Georgieva and Ioana Giurgiu and Serge Monney and Haris Pozidis and Viviane Potocnik and Mitch Gusat},
  title = {{CCA: An ML Pipeline for Cloud Anomaly Troubleshooting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21716},
}
Figure