TL;DR
This paper introduces a novel approach to detecting anomalies and their root causes in multivariate time series by leveraging causal inference. Instead of just modeling the joint distribution of data, it learns the underlying causal relationships and treats anomalies as deviations from these local causal mechanisms, leading to more accurate detection and easier root cause identification.
What Problem Does This Paper Address?
Monitoring complex systems like IT infrastructure or manufacturing plants relies heavily on analyzing multivariate time series data. Detecting anomalies (unexpected behavior) and pinpointing their root causes is crucial for maintaining system health and availability. Traditional methods often struggle because they either treat each time series independently, ignoring crucial interdependencies, or model the overall joint distribution, which can be computationally expensive and obscure the specific mechanisms causing anomalies. This makes it difficult to pinpoint why an anomaly occurred, hindering effective remediation.
Methods
The core of this paper's contribution is a causality-based framework for anomaly detection and root cause analysis. Here's how it works:
- Causal Discovery: The first step is to learn the causal structure of the multivariate time series. This is done using established causal discovery algorithms (like PC or FGES) on observational data to infer a directed acyclic graph (DAG) representing causal relationships between variables.
- Local Causal Mechanisms: The learned causal graph decomposes the complex multivariate system into simpler, independent "modules." Each module represents a local causal mechanism, defined by the conditional probability distribution of a variable given its causal parents. These distributions are learned using standard machine learning models (e.g., CVAE, MLP).
- Anomaly Detection: An instance (a data point at a specific time) is flagged as anomalous if any of its local causal mechanisms are violated. This means the observed value of a variable doesn't follow its expected distribution given its causal parents. The anomaly score is derived from the minimum probability across all local mechanisms.
- Root Cause Identification: Because the problem is decomposed into independent local mechanisms, identifying the root cause becomes straightforward. If a local mechanism is violated, the variable whose mechanism is broken is a direct root cause. A PageRank-like algorithm is used to propagate and refine these root cause scores.
Datasets
The authors evaluated their approach on:
- Simulated Datasets: Generated with varying linear/nonlinear relationships and different anomaly types (measurement, intervention, effect, mixed discrete/continuous).
- Public Datasets:
- Secure Water Treatment (SWaT)
- Water Distribution (WADI)
- Soil Moisture Active Passive (SMAP)
- Mars Science Laboratory (MSL)
- Real-world AIOps Application: A dataset from a database service monitoring system, comprising 61 time series variables.
Key Insights and Ideas
- Causality as a Lens for Anomalies: The paper reframes anomalies not just as statistical outliers, but as deviations from the underlying causal process that generates the data. This perspective is key to understanding why anomalies occur.
- Modularity for Simplicity and Interpretability: By leveraging the modularity of causal systems, the complex multivariate problem is broken down into smaller, manageable, low-dimensional anomaly detection tasks. This significantly simplifies detection and directly points to root causes.
- Root Cause Analysis is Intrinsic, Not an Add-on: Unlike many existing methods where root cause analysis is a separate, post-hoc step, this causal framework integrates it naturally. When an anomaly is detected in a local mechanism, the variable involved is a direct candidate for a root cause.
- Robustness to Causal Graph Imperfections: The experiments suggest that the approach is relatively robust to inaccuracies in the learned causal graph, meaning it doesn't need a perfectly accurate graph to perform well.
Key Results
- Superior Anomaly Detection Performance: The proposed causality-based method consistently outperformed several state-of-the-art unsupervised anomaly detection methods (e.g., IF, AE, VAE, LSTM-VAE, DAGMM, OmniAnomaly, USAD, GANF) across simulated and public datasets, achieving significantly higher F1 scores, especially in challenging nonlinear scenarios. For instance, on simulated "nonlinear, intervention" data, their F1 score was 0.759, compared to the best baseline of 0.589.
- Accurate Root Cause Identification: The method demonstrated strong performance in root cause analysis, achieving high Top-k hit ratios (e.g., HR@3 >= 0.95 on simulated datasets), significantly outperforming a baseline that ignored causal relationships.
- Practical Feasibility: The approach was successfully applied to a real-world AIOps case study, correctly identifying anomalies and pinpointing root causes related to I/O issues, validated by domain experts.
- Efficiency: While causal discovery can be time-consuming, the anomaly detection and root cause identification steps are fast, making it suitable for real-time applications.
Limitations and Future Work
- Training Anomalies: The paper acknowledges that anomalies in the training data can negatively impact causal discovery and anomaly detection performance. They propose an iterative solution to handle this, but note that for practical applications with few training anomalies, a single iteration is often sufficient.
- Causal Discovery Assumptions: The performance relies on the quality of the learned causal graph. While robust to some inaccuracies, significant errors could still impact results. Future work could explore more robust causal discovery methods or ways to explicitly handle uncertainty in the graph.
- Scalability of Causal Discovery: While the anomaly detection part is efficient, causal discovery itself can be computationally intensive for very high-dimensional data.
- Future Work: The authors suggest exploring more advanced causal discovery techniques and applying the framework to a wider range of real-world systems.
Summary updated