Simple Root Cause Analysis by Separable Likelihoods

Maciej SkorskiarXiv 2018 · OpenReview

TL;DR

This paper introduces a novel framework for Root Cause Analysis (RCA) of anomalies in complex systems, balancing accuracy with interpretable explanations. By leveraging a "separability" assumption on the predictive posterior of Bayesian models, the authors provide a method to pinpoint which features contribute most to an anomaly, making it easier to diagnose and fix issues in real-world applications like server error logs.

What Problem Does This Paper Address?

In industrial settings, identifying anomalies is only half the battle; understanding why an anomaly occurred (Root Cause Analysis or RCA) is crucial for quick resolution. Traditional anomaly detection methods, especially those using complex probabilistic models, often provide highly accurate anomaly scores but lack interpretability. This makes it difficult for users to understand the underlying causes. The challenge lies in bridging this gap: how can we achieve both accurate anomaly detection and provide clear, actionable explanations for the anomalies, especially in systems with diverse data types (continuous and categorical) and interdependencies?

Methods

The core contribution is a framework for RCA based on a "separability" property of the predictive posterior in Bayesian models. This property arises when the Hessian matrix of the log-likelihood at the mode is diagonal. This condition allows the authors to decompose the anomaly score (specifically, the deficiency relative to the mode) into contributions from individual features.

The key steps are:

  1. Model Selection: The framework is designed to work with models that satisfy the separability condition. The paper demonstrates this for Dirichlet-Multinomial (DM) and Bernoulli Naive Bayes (BNB) models, which are common for count data and text classification, respectively.
  2. Laplace Approximation: To approximate the predictive posterior and derive the separability property, the authors employ Laplace approximation. This involves expanding the log-likelihood around its mode and focusing on the quadratic terms, which naturally lead to a diagonal Hessian under the separability assumption.
  3. Feature Impact Calculation: Once separability is established, the paper derives formulas (e.g., Corollary 1 for DM, Corollary 2 for BNB) that quantify the "impact" of each feature on the anomaly score. These impacts are essentially the contributions of individual features to the overall deficiency from the mode.
  4. Bayesian Network Integration: The framework is integrated into a Tree-Augmented Network (TAN) Bayesian Network to model the dependencies between different features in a real-world scenario. This allows for a structured approach to RCA across multiple interacting variables.

Datasets

The paper uses a real-world dataset of web server error logs. This dataset contains information on the date, region, project, procedure, error detail, and error count. The data was collected over more than 120 consecutive days. No new benchmarks are introduced; the focus is on applying the framework to existing real-world data.

Key Insights and Ideas

  • Separability as a Key to Interpretability: The paper highlights that a specific mathematical property (diagonal Hessian at the mode) of the predictive posterior, termed "separability," is the key to achieving interpretable RCA. This property allows for a clean decomposition of anomaly scores into feature-specific contributions.
  • Deficiency from the Mode is the Anomaly Signal: Instead of just looking at low likelihood, the framework focuses on the deficiency of a data point's log-likelihood relative to the mode of the posterior. This difference is what gets decomposed, providing a more focused explanation of why a point is anomalous.
  • Practical Applicability to Common Models: The framework is shown to be applicable to widely used models like Dirichlet-Multinomial (for counts) and Bernoulli Naive Bayes (for text classification), making it relevant for many practical applications.
  • Hierarchical RCA: By embedding the models within a Bayesian Network (specifically a TAN), the paper demonstrates how to perform RCA at different levels of a system's hierarchy (e.g., project, procedure, error message), providing a comprehensive view of the anomaly's origin.

Key Results

The paper demonstrates the framework's effectiveness through a case study on web server error logs.

  • Anomaly Detection: The framework successfully identifies anomalous days in the server logs.
  • Project-Level RCA: For anomalies related to project counts, Corollary 1 is used to identify "most impacting projects," showing that anomalies often correspond to peaks in project hits.
  • Procedure and Error Message RCA: For anomalies in procedures and error messages, Corollary 2 (BNB) is applied. This identifies keywords with the biggest negative influence on the likelihood. For example, for an anomaly on 2018/05/17 in EMEA, keywords like {'object', 'set', 'reference', 'instance', 'connection'} were identified as highly influential. By inspecting these keywords, the authors could pinpoint specific procedures and error messages (e.g., "Object reference not set to an instance of an object") that were significantly different on anomaly days compared to reference days.
  • Qualitative Explanations: The framework provides concrete, human-readable explanations by identifying specific keywords, procedures, and error messages associated with anomalies, which is the primary goal of RCA.

Limitations and Future Work

  • Separability Assumption: The core limitation is the reliance on the "separability" assumption, which requires the Hessian at the mode to be diagonal. While shown to hold for several important models, it might not be universally applicable to all complex Bayesian models.
  • Approximation Accuracy: The use of Laplace approximation is an approximation. The accuracy of the RCA might depend on how well this approximation captures the true posterior, especially in highly complex or multimodal posteriors.
  • Future Work: The paper suggests that the framework can be extended to other models that satisfy the separability condition. Further exploration into how to relax the separability assumption or handle cases where it doesn't hold could be a valuable direction. The authors also imply that the framework's ability to provide "reasonable and simple explanations" is a key advantage for future industrial adoption.

Summary updated

Abstract

Root Cause Analysis for Anomalies is challenging because of the trade-off between the accuracy and its explanatory friendliness, required for industrial applications. In this paper we propose a framework for simple and friendly RCA within the Bayesian regime under certain restrictions (that Hessian at the mode is diagonal, here referred to as \emph{separability}) imposed on the predictive posterior. We show that this assumption is satisfied for important base models, including Multinomal, Dirichlet-Multinomial and Naive Bayes. To demonstrate the usefulness of the framework, we embed it into the Bayesian Net and validate on web server error logs (real world data set).
BibTeX
@misc{skorski2018simple,
  author = {Maciej Skorski},
  title = {{Simple Root Cause Analysis by Separable Likelihoods}},
  year = {2018},
  eprint = {1808.04302},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.04302},
}
Figure