Learning Dependencies in Distributed Cloud Applications to Identify and Localize Anomalies

Dominik Scheinert, Alexander Acker, Lauritz Thamsen, Morgan K. Geldenhuys, Odej KaoarXiv 2021 · DOI · arXiv · OpenReview

TL;DR

This paper introduces Arvalus and D-Arvalus, novel graph-based machine learning methods for identifying and pinpointing anomalies in complex distributed cloud applications. By modeling system components and their dependencies as a graph, D-Arvalus leverages graph convolutions to significantly improve anomaly detection and localization accuracy compared to methods that ignore these interdependencies.

What Problem Does This Paper Address?

Operating large-scale distributed cloud applications, like microservice architectures, is incredibly challenging. As these systems grow to encompass thousands of interconnected components, identifying and fixing problems (anomalies) becomes a monumental task. When an anomaly occurs in one component, it can ripple through the system, triggering a cascade of alerts that make it difficult for human operators to find the original source of the issue. Existing anomaly detection methods often struggle with this propagation, and traditional root cause analysis techniques can be overly reliant on manual rules, specific metrics, or access to internal system details. This paper aims to bridge this gap by providing a data-driven approach that automatically learns and utilizes the complex web of dependencies within these systems to improve both anomaly detection and localization.

Methods

The core of the proposed solution lies in representing the distributed cloud application as a graph. Each system component (e.g., a virtual machine, a microservice) is a node, and the dependencies between them are represented as edges. The approach, named Arvalus, has two main variants:

  • Arvalus: This variant focuses on extracting features from Key Performance Indicator (KPI) time series data for each component. These features are then used directly to predict whether a component is experiencing a normal state or an anomaly.
  • D-Arvalus: This is an enhanced version that builds upon Arvalus by incorporating graph convolution (GC). D-Arvalus uses the graph structure and the learned features of neighboring nodes to enrich the feature representation of each node. This is achieved by aggregating information from connected nodes, effectively learning how dependencies influence a component's state.

The process involves:

  1. Node Feature Extraction: Raw KPI data for each component is transformed into a meaningful feature vector using convolutional layers, activation functions, and normalization techniques.
  2. Graph Construction: An initial graph is built using meta-information about the system deployment. Edges are defined with specific types (e.g., identity, host-guest, dependency) and are further processed to learn trainable edge weights.
  3. Graph Convolution (D-Arvalus): The learned node features are combined with the graph structure and edge weights to aggregate information from neighboring nodes. This process allows the model to understand how the state of one component is influenced by others.
  4. Anomaly Identification and Localization: The final node features (either raw or enriched by GC) are used to classify each component as normal or anomalous. For localization, the model aims to pinpoint the specific component(s) responsible for the anomaly.

Datasets

The paper does not use publicly available datasets. Instead, the authors simulate a distributed cloud application environment. This simulation is based on the deployment structures of real cloud applications, including a virtual IMS and an RTMP content streaming service. They synthetically inject two types of anomalies into the simulated system components, generating labeled data for training and evaluation. The simulation involves 51 system components, each with 10 KPIs.

Key Insights and Ideas

  • Dependencies Matter: The most significant insight is that explicitly modeling and learning the interdependencies between system components is crucial for effective anomaly detection and localization in distributed systems. Ignoring these relationships leads to a significant performance drop.
  • Graph Convolution as a Powerful Tool: Graph convolution provides a principled way to aggregate information from a component's neighborhood, allowing the model to capture the propagation of anomalies and the influence of upstream or downstream services.
  • D-Arvalus Outperforms Arvalus: The variant that incorporates graph convolution (D-Arvalus) consistently shows superior performance in both anomaly identification and localization compared to the baseline Arvalus. This highlights the value of the learned dependency information.
  • Improved Localization Precision: D-Arvalus not only identifies anomalies better but also pinpoints their origin with greater accuracy, reducing the search space for human operators.

Key Results

The evaluation demonstrates the effectiveness of D-Arvalus:

  • Anomaly Detection: D-Arvalus achieved a macro F1 score of 0.967 for binary anomaly detection, compared to 0.925 for Arvalus. For multiclass anomaly identification, D-Arvalus reached 0.954, while Arvalus scored 0.896.
  • Anomaly Localization: D-Arvalus showed a significant improvement in localization performance, with an 11% higher Mean Average Precision (MAP) score compared to Arvalus. Specifically, D-Arvalus achieved 99.9% accuracy for "Local" anomalies, 99.0% for "Adversary" anomalies, and 97.6% for "Neighborhood" anomalies, outperforming Arvalus in all scenarios.

Limitations and Future Work

  • Simulated Environment: The evaluation was conducted on a simulated environment. The authors acknowledge that real-world cloud deployments can be more complex and dynamic.
  • Future Work: The authors plan to evaluate their method using data from actual real-world cloud deployments. They also aim to explore further improvements and applications of their graph-based approach.

Summary updated

Abstract

Operation and maintenance of large distributed cloud applications can quickly become unmanageably complex, putting human operators under immense stress when problems occur. Utilizing machine learning for identification and localization of anomalies in such systems supports human experts and enables fast mitigation. However, due to the various inter-dependencies of system components, anomalies do not only affect their origin but propagate through the distributed system. Taking this into account, we present Arvalus and its variant D-Arvalus, a neural graph transformation method that models system components as nodes and their dependencies and placement as edges to improve the identification and localization of anomalies. Given a series of metric KPIs, our method predicts the most likely system state - either normal or an anomaly class - and performs localization when an anomaly is detected. During our experiments, we simulate a distributed cloud application deployment and synthetically inject anomalies. The evaluation shows the generally good prediction performance of Arvalus and reveals the advantage of D-Arvalus which incorporates information about system component dependencies.
BibTeX
@misc{scheinert2021learning,
  author = {Dominik Scheinert and Alexander Acker and Lauritz Thamsen and Morgan K. Geldenhuys and Odej Kao},
  title = {{Learning Dependencies in Distributed Cloud Applications to Identify and Localize Anomalies}},
  year = {2021},
  eprint = {2103.05245},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.05245},
}
Figure