Fault Location Method Based on Dynamic Operation and Maintenance Map and Common Alarm Points Analysis

Sheng Wu, Jihong GuanAlgorithms 2024 · OpenReview

TL;DR

This paper introduces a novel fault location method for complex distributed systems that leverages dynamic operation maps and common alarm point analysis. By constructing a real-time map of system dependencies and analyzing alarm patterns, the method efficiently pinpoints the root cause of failures, achieving over 82% accuracy with analysis times under 2 minutes.

What Problem Does This Paper Address?

In today's massive and interconnected distributed information systems, pinpointing the exact infrastructure component causing an application error is a significant challenge. The sheer scale of components (applications, OS, databases, servers, networks) and their intricate relationships create a "silo effect," where monitoring systems operate independently, hindering effective troubleshooting. Existing methods like knowledge graphs are complex and often limited to specific domains, while indicator correlation analysis lacks crucial configuration details. This leaves a gap in accurately and efficiently identifying the root cause of faults, especially under strict uptime requirements like those in financial systems.

Methods

The proposed method comprises three main modules:

  1. Dynamic Operation and Maintenance Map Construction: This module builds a real-time, graph-based representation of the system's topology. It integrates data from various sources (CMDB, cloud platforms, network probes) to model nodes (applications, servers, networks) and their relationships (access, deployment, dependency). This dynamic map is crucial for understanding the system's current state and interconnections.
  2. Multi-Source Alarm Screening and Feature Extraction: This module addresses the "alarm storm" problem during failures. It uses a sliding window approach to filter relevant alarms and then extracts key features, focusing on IP addresses and related operational entities. This step aims to reduce noise and highlight critical alarm information.
  3. Map Addressing and Common Point Analysis: This is the core fault localization engine. It queries the dynamic operation map using alarm IPs to enrich the alarm data with topological context. Then, it performs commonality analysis on these enriched alarms, identifying "common points" across deployment relationships, connections (physical/logical), and access flows. This analysis, inspired by word frequency statistics, ranks potential root cause IPs based on their occurrence frequency.

The method is designed to be more concise than knowledge graph construction and more accurate than indicator correlation analysis by incorporating configuration correlation information.

Datasets

The study utilized a dataset from a bank's key personal settlement business system. This dataset comprised centralized monitoring alert events from a 15-minute period preceding nine simulated infrastructure failures (network, system, database, and storage). The authors did not introduce a new benchmark dataset but rather used real-world operational data from a production environment.

Key Insights and Ideas

  • Dynamic Maps are Key: The paper strongly emphasizes that a dynamic and real-time representation of system topology is essential for accurate fault localization, moving beyond static configurations.
  • Alarm Filtering is Crucial for Performance: The "alarm storm" problem significantly impacts troubleshooting speed. Effective filtering and screening of alarms are critical for reducing processing load and improving timeliness.
  • Commonality Analysis Bridges Gaps: By analyzing commonalities in deployment, connection, and access flows derived from the dynamic map and alarms, the method can effectively bridge the information gaps left by purely indicator-based or static knowledge graph approaches.
  • Simplicity Over Complexity: The authors highlight that their approach avoids the complex construction process of knowledge graphs, making it more efficient and practical for real-world deployment.

Key Results

The proposed method demonstrated a fault hit rate exceeding 82% in practical validation. The analysis time for locating a fault was consistently less than 2 minutes. In one experiment, the alarm screening module achieved an alarm compression ratio of 3.99%, reducing time consumption by 96% (from 428s to 17s). The experiments also confirmed that the accuracy of fault localization is directly dependent on the completeness and type of dynamic map topology data provided.

Limitations and Future Work

  • Data Availability: The paper states that the datasets presented are not readily available as they are from a third-party business.
  • Dynamic Map Completeness: The success of the method is directly tied to the completeness of the dynamic map's topology data. If critical topological information is missing, fault localization accuracy can be compromised.
  • Future Work: While not explicitly detailed, the success of the method suggests potential for further refinement in alarm feature extraction, more sophisticated commonality analysis algorithms, and broader application across different types of distributed systems. The paper's focus on a bank's payment system indicates a strong potential for adoption in other critical infrastructure environments.

Summary updated

Abstract

Under a distributed information system, the scale of various operational components such as applications, operating systems, databases, servers, and networks is immense, with intricate access relationships. The silo effect of each professional is prominent, and the linkage mechanism is insufficient, making it difficult to locate the infrastructure components that cause exceptions under a particular application. Current research only plays a role in local scenarios, and its accuracy and generalization are still very limited. This paper proposes a novel fault location method based on dynamic operation maps and alarm common point analysis. During the fault period, various alarm entities are associated with dynamic operation maps, and alarm common points are obtained based on graph search addressing methods, covering deployment relationship common points, connection common points (physical and logical), and access flow common points. This method, compared with knowledge graph approaches, eliminates the complex process of knowledge graph construction, making it more concise and efficient. Furthermore, in contrast to indicator correlation analysis methods, this approach supplements with configuration correlation information, resulting in more precise positioning. Through practical validation, its fault hit rate exceeds 82%, which is significantly better than the existing main methods.
BibTeX
@inproceedings{wu2024fault,
  author = {Sheng Wu and Jihong Guan},
  title = {{Fault Location Method Based on Dynamic Operation and Maintenance Map and Common Alarm Points Analysis}},
  booktitle = {Algorithms},
  year = {2024},
  url = {https://openreview.net/forum?id=jit7NX0Wk8},
}
Figure