Abstract
Detecting anomalies and the corresponding root causes in multivariate time series plays an important role in monitoring the behaviors of various real-world systems, e.g., IT system operations or manufacturing industry. Previous anomaly detection approaches model the joint distribution without considering the underlying mechanism of multivariate time series, making them computationally hungry and hard to identify root causes. In this paper, we formulate the anomaly detection problem from a causal perspective and view anomalies as instances that do not follow the regular causal mechanism to generate the multivariate data. We then propose a causality-based framework for detecting anomalies and root causes. It first learns the causal structure from data and then infers whether an instance is an anomaly relative to the local causal mechanism whose conditional distribution can be directly estimated from data. In light of the modularity property of causal systems (the causal processes to generate different variables are irrelevant modules), the original problem is divided into a series of separate, simpler, and low-dimensional anomaly detection problems so that where an anomaly happens (root causes) can be directly identified. We evaluate our approach with both simulated and public datasets as well as a case study on real-world AIOps applications, showing its efficacy, robustness, and practical feasibility.
Introduction
In complex systems like IT infrastructure or manufacturing plants, monitoring hundreds of variables (KPIs) simultaneously is standard practice. While detecting when a system enters an abnormal state is critical, identifying the "root cause"—the specific component that failed—is often the more difficult and valuable task.
Traditional multivariate deep learning methods typically attempt to model the joint distribution of all variables at once. While powerful, these "black box" models are computationally expensive and struggle to explain how a failure propagates through a system. This paper introduces a framework that moves away from joint distribution modeling to focus on the underlying causal mechanisms of the data-generating process.
Methodology: A Causal Perspective
The core innovation of this framework is treating an anomaly as an instance that violates the regular causal mechanism of the system. By decomposing a complex multivariate system into modular, local causal relationships, the problem becomes more manageable and interpretable.
The approach follows three main steps:
- Causal Discovery: The algorithm learns the structure of the system (a Directed Acyclic Graph or DAG) from historical data. This identifies which variables influence others (e.g., how CPU usage impacts temperature).
- Modular Decomposition: Leveraging the modularity property of causal systems, the joint distribution is factored into a series of simpler, local conditional distributions: $P(x_i | \text{Parents of } x_i)$. This transforms one high-dimensional problem into many low-dimensional ones.
- Anomaly Detection & Root Cause Analysis (RCA):
- A Conditional Variational Autoencoder (CVAE) is trained for each variable to learn its normal behavior given its causal parents.
- An anomaly score is calculated based on deviations from these local mechanisms.
- RCA: If a local mechanism is violated, that variable is flagged as a candidate root cause. A PageRank-inspired algorithm then propagates these scores through the causal graph to distinguish the "source" of the problem from its downstream "symptoms."
Evaluation and Results
The framework was evaluated using simulated datasets (linear and nonlinear), standard public benchmarks (SWaT, WADI, SMAP, MSL), and a real-world AIOps case study monitoring database service KPIs.
Key Findings
- Superior Performance: On simulated nonlinear data with "intervention" anomalies, the proposed method achieved an F1-score of 0.759, significantly outperforming baselines like DAGMM (0.589) and OmniAnomaly (0.545).
- Accurate Root Cause Identification: The model achieved a Top-3 Hit Ratio (HR@3) of ≥ 0.95 for linear settings and ≥ 0.75 for nonlinear settings.
- Real-World Feasibility: In the AIOps case study, the model correctly identified I/O-related issues as root causes for database incidences, matching expert domain feedback.
- Robustness: The method remains effective even if the initial causal discovery step produces an imperfect graph, provided the major dependencies are captured.
Conclusion and Future Work
By breaking a system into independent causal modules, anomaly detection becomes more robust and root cause identification becomes a direct output of the model logic. While the current framework assumes a Directed Acyclic Graph (DAG) and system stationarity, future research will explore adapting the approach for cyclic graphs (feedback loops) and highly dynamic environments where causal relationships may shift over time.
Summary updated