Operation and maintenance analysis platform at IHEP

Qingbao Hu, Lu Wang, Wei Zheng, Xiaowei Jiang2022 · DOI

Abstract

As the scale of computing facilities continues to grow and the computing environment becomes more and more complex, the difficulty of operation and maintenance of large-scale computing clusters is also increasing. Operation and maintenance methods based solely on configuration management automation technology cannot quickly and effectively solve various service failures in computing clusters. It is urgent to adopt emerging technologies to obtain comprehensive cluster operation and maintenance information, integrate monitoring data from multiple heterogeneous sources, and comprehensively analyze anomalous patterns in monitoring data. Based on the results of data analysis, it becomes possible to locate the root cause of service failures and help computing clusters quickly restore services. To provide a more stable cluster operating environment, IHEPCC combined big data technology and data analysis index tools to design and implement the operation and maintenance analysis toolkit (OMAT) as an open framework, which includes data collection, correlation analysis, anomaly detection, and alerting and other functions. This report introduces the architecture, processing capabilities, and some key functions of OMAT. Combined with the processing flow of monitoring data, it introduces the specific implementation of the system in data collection, data processing, data storage, and data visualization. The current OMAT platform has been applied to multiple cross-regional computing clusters including IHEP, covering about 5000 nodes.

Introduction

The rapid expansion of scientific computing facilities, driven by experiments such as BESIII and JUNO, has introduced significant complexity to cluster operation and maintenance (O&M). Traditional O&M methods, often relying on automated configuration management alone, struggle to address service failures in large-scale, heterogeneous environments. A primary challenge is the existence of "isolated islands" of monitoring data, where tools like Nagios, Ganglia, and Zabbix operate independently, preventing effective event correlation. To bridge these gaps, researchers at the Institute of High Energy Physics (IHEP) developed the Operation and Maintenance Analysis Toolkit (OMAT).

System Architecture: The OMAT Framework

OMAT is designed as an open framework that leverages big data technologies to provide a unified "brain" for cluster management. Its architecture is organized into four functional layers:

  1. Ecological Integration Layer: Reuses existing monitoring tools (Zabbix, Syslog-ng, etc.) as data sources, acting as the primary sensors for the system.
  2. Data Processing Layer: Utilizes Apache Kafka for high-speed caching and Spark Streaming for real-time analysis. It employs machine learning algorithms, such as Isolation Forest for unsupervised anomaly detection and K-means for clustering, to identify complex failure patterns without manual thresholds.
  3. Data Warehouse Layer: Implements a tiered storage strategy using Elasticsearch for logs and InfluxDB for time-series metrics. A "hot/warm/cold" lifecycle moves data from SSDs to SAS drives and eventually to tape to optimize costs.
  4. Application & Configuration Layer: Provides a Django-based API for custom dashboards and automated decision-making based on expert experience.

Data Integration and Multi-Site Monitoring

OMAT currently manages a cross-regional infrastructure comprising approximately 5,000 nodes and over 77,000 CPU cores. The platform integrates data from diverse sites, including Beijing, Dongguan, and various university partners, covering node status, storage performance, and user behavior.

Figure 4: Dashboard monitoring multiple data centers, showing CPU resources, utilization, and storage capacity across sites like IHEPCC and DongGuan.

Intelligent Anomaly Detection and Automated Management

A key innovation of OMAT is its shift from passive monitoring to active management. By tagging raw data with metadata (e.g., physical location or specific job IDs), the system provides actionable context. For example, unsupervised learning models can identify "greedy" I/O patterns that degrade shared storage performance even when technical limits aren't breached.

The platform's real-time capabilities are demonstrated through automated fault recovery. When OMAT detects a node-level anomaly, such as a mount point failure, it can automatically trigger policy updates in the HTCondor scheduler to isolate the affected node, thereby protecting ongoing scientific jobs from cascading failures.

Figure 5: Anomaly detection interface showing the count of anomalous nodes and specific failure details for devices like bws0950, which are then used to inform scheduler adjustments.

Conclusion

OMAT has successfully unified IHEP's monitoring landscape, providing a scalable solution for managing 5,000 nodes across multiple regions. By moving beyond simple threshold alerts to machine-learning-driven anomaly detection, the platform ensures higher stability for high-energy physics computing. Future work will focus on expanding the library of machine learning algorithms and further integrating business logic into the automated decision-making process to improve overall system reliability.

Summary updated

Abstract

As the scale of computing facilities continues to grow and the computing environment becomes more and more complex, the difficulty of operation and maintenance of large-scale computing clusters is also increasing. Operation and maintenance methods based solely on configuration management automation technology cannot quickly and effectively solve various service failures in computing clusters. It is urgent to adopt emerging technologies to obtain comprehensive cluster operation and maintenance information, integrate monitoring data from multiple heterogeneous sources, and comprehensively analyze anomalous patterns in monitoring data. Based on the results of data analysis, it becomes possible to locate the root cause of service failures and help computing clusters quickly restore services. To provide a more stable cluster operating environment, IHEPCC combined big data technology and data analysis index tools to design and implement the operation and maintenance analysis toolkit (OMAT) as an open framework, which includes data collection, correlation analysis, anomaly detection, and alerting and other functions. This report introduces the architecture, processing capabilities, and some key functions of OMAT. Combined with the processing flow of monitoring data, it introduces the specific implementation of the system in data collection, data processing, data storage, and data visualization. The current OMAT platform has been applied to multiple cross-regional computing clusters including IHEP, covering about 5000 nodes. The collected information includes node status, storage performance, network traffic, user operations, account security, power environment, and other operation and maintenance indicators to ensure the computing cluster’s performance and stable operation.
BibTeX
@misc{hu2022operation,
  author = {Qingbao Hu and Lu Wang and Wei Zheng and Xiaowei Jiang},
  title = {{Operation and maintenance analysis platform at IHEP}},
  year = {2022},
  doi = {10.22323/1.415.0011},
}

Loading related content...

Figure