TL;DR
This paper introduces a novel fault location method for complex distributed systems that leverages dynamic operation maps and common alarm point analysis. By constructing a real-time map of system dependencies and analyzing alarm patterns, the method efficiently pinpoints the root cause of failures, achieving over 82% accuracy with analysis times under 2 minutes.
What Problem Does This Paper Address?
In today's massive and interconnected distributed information systems, pinpointing the exact infrastructure component causing an application error is a significant challenge. The sheer scale of components (applications, OS, databases, servers, networks) and their intricate relationships create a "silo effect," where monitoring systems operate independently, hindering effective troubleshooting. Existing methods like knowledge graphs are complex and often limited to specific domains, while indicator correlation analysis lacks crucial configuration details. This leaves a gap in accurately and efficiently identifying the root cause of faults, especially under strict uptime requirements like those in financial systems.
Methods
The proposed method comprises three main modules:
- Dynamic Operation and Maintenance Map Construction: This module builds a real-time, graph-based representation of the system's topology. It integrates data from various sources (CMDB, cloud platforms, network probes) to model nodes (applications, servers, networks) and their relationships (access, deployment, dependency). This dynamic map is crucial for understanding the system's current state and interconnections.
- Multi-Source Alarm Screening and Feature Extraction: This module addresses the "alarm storm" problem during failures. It uses a sliding window approach to filter relevant alarms and then extracts key features, focusing on IP addresses and related operational entities. This step aims to reduce noise and highlight critical alarm information.
- Map Addressing and Common Point Analysis: This is the core fault localization engine. It queries the dynamic operation map using alarm IPs to enrich the alarm data with topological context. Then, it performs commonality analysis on these enriched alarms, identifying "common points" across deployment relationships, connections (physical/logical), and access flows. This analysis, inspired by word frequency statistics, ranks potential root cause IPs based on their occurrence frequency.
The method is designed to be more concise than knowledge graph construction and more accurate than indicator correlation analysis by incorporating configuration correlation information.
Datasets
The study utilized a dataset from a bank's key personal settlement business system. This dataset comprised centralized monitoring alert events from a 15-minute period preceding nine simulated infrastructure failures (network, system, database, and storage). The authors did not introduce a new benchmark dataset but rather used real-world operational data from a production environment.
Key Insights and Ideas
- Dynamic Maps are Key: The paper strongly emphasizes that a dynamic and real-time representation of system topology is essential for accurate fault localization, moving beyond static configurations.
- Alarm Filtering is Crucial for Performance: The "alarm storm" problem significantly impacts troubleshooting speed. Effective filtering and screening of alarms are critical for reducing processing load and improving timeliness.
- Commonality Analysis Bridges Gaps: By analyzing commonalities in deployment, connection, and access flows derived from the dynamic map and alarms, the method can effectively bridge the information gaps left by purely indicator-based or static knowledge graph approaches.
- Simplicity Over Complexity: The authors highlight that their approach avoids the complex construction process of knowledge graphs, making it more efficient and practical for real-world deployment.
Key Results
The proposed method demonstrated a fault hit rate exceeding 82% in practical validation. The analysis time for locating a fault was consistently less than 2 minutes. In one experiment, the alarm screening module achieved an alarm compression ratio of 3.99%, reducing time consumption by 96% (from 428s to 17s). The experiments also confirmed that the accuracy of fault localization is directly dependent on the completeness and type of dynamic map topology data provided.
Limitations and Future Work
- Data Availability: The paper states that the datasets presented are not readily available as they are from a third-party business.
- Dynamic Map Completeness: The success of the method is directly tied to the completeness of the dynamic map's topology data. If critical topological information is missing, fault localization accuracy can be compromised.
- Future Work: While not explicitly detailed, the success of the method suggests potential for further refinement in alarm feature extraction, more sophisticated commonality analysis algorithms, and broader application across different types of distributed systems. The paper's focus on a bank's payment system indicates a strong potential for adoption in other critical infrastructure environments.
Summary updated