Introduction
In modern Artificial Intelligence for IT Operations (AIOps), the scale and complexity of cloud infrastructures present significant challenges for system reliability. A single microservices-based platform may monitor thousands of Key Performance Indicators (KPIs). When performance degradation occurs, Site Reliability Engineers (SREs) are often overwhelmed by a "metric storm," making manual root cause identification labor-intensive and error-prone.
PyRCA is an open-source Python library developed by Salesforce AI to address these challenges. It provides a unified, end-to-end framework for metric-based Root Cause Analysis (RCA), bridging the gap between theoretical causal inference and practical system monitoring.
The RCA Framework
The PyRCA framework streamlines the transition from anomaly detection to incident resolution. As illustrated in the framework overview, the process begins with cloud monitoring systems collecting metrics from various services. When an anomaly is detected, the framework constructs a dependency graph—leveraging both data-driven causal discovery and expert knowledge—to localize the root cause for the SRE.
Library Architecture
PyRCA is designed with a modular three-layer architecture to ensure flexibility and ease of use for both researchers and practitioners, as shown in the architecture diagram below.
- Input Layer: Handles diverse data sources, including metric time-series (via pandas DataFrames), model configurations (YAML), and domain-specific expert knowledge (YAML).
- Model Layer: Contains the core algorithmic components, including anomaly detectors, dependency graph constructors, and root cause analyzers.
- Output Layer: Provides actionable results through model evaluation metrics and an interactive GUI dashboard for visualization.
Methodological Paradigms
PyRCA categorizes RCA approaches into two primary paradigms:
Two-Phase Approaches
This paradigm separates the problem into two distinct stages:
- Causal Discovery: The library builds a causal graph representing metric influences. It supports constraint-based algorithms like PC and score-based algorithms like GES.
- Root Cause Scoring: Once the graph is established, algorithms such as Random Walk and Hypothesis Testing (HT) trace anomalies through the graph to identify the most likely root causes.
One-Phase Approaches
For massive systems where full graph construction is computationally prohibitive, PyRCA includes one-phase models like $ε$-Diagnosis and RCD. These methods bypass explicit graph construction to directly localize root causes from abnormal data.
A key innovation in PyRCA is the integration of Expert Knowledge. Users can define "forced" or "forbidden" causal links in a configuration file, allowing human intuition to refine machine-learned models.
Evaluation and Results
The library was evaluated using a simulated dataset mimicking complex system dependencies, consisting of 500 graphs with 20 nodes each. Key findings include:
- Algorithm Performance: When the true causal graph is known, Hypothesis Testing (HT) achieved a Recall@1 of 1.00, outperforming other methods.
- Causal Discovery Accuracy: The PC algorithm demonstrated superior performance in discovery tasks, achieving an F1 score of 0.78, compared to 0.45 for GES.
- Graph Sensitivity: Results indicate a performance drop when using estimated graphs versus ground-truth topologies, highlighting causal discovery as a critical bottleneck in automated RCA.
Conclusion and Future Work
PyRCA provides a standardized interface for applying advanced causal inference to system monitoring. While currently focused on numerical metrics, future development aims to integrate unstructured data such as logs and traces. By lowering the barrier to entry for sophisticated RCA methods, PyRCA empowers SREs to resolve incidents with greater speed and precision.
Summary updated