A Blueprint of Log Based Monitoring and Diagnosing Framework in Large Distributed Environments

Yining Zhao, Xiaodong Wang, Haili Xiao, Xuebin Chi2019 · DOI

Abstract

Distributed systems have kept scaling upward since this concept appears, and they soon evolve to environments that contain heterogeneous components playing different roles, making it difficult to understand how the large environment works or if any undesired matters happened from security point of view. Logs, produced by devices, sub-systems and running processes, are a very important source to help system maintainers to get relative security knowledge. But there are too many logs and too many kinds of logs to deal with, which makes manual checking impossible. In this work we will share some of our experiences in log processing and analyzing. We have summarized some common major steps that appear in most of the existing log analysis approaches, including log selection, log classification, information analyses and result feedback. We also represent a general framework that monitors events, analyzes hidden information and diagnoses the healthy state for large distributed computing environments bases on logs. Although the framework we initially designed was for the maintenance for CNGrid, its process is adaptable to other distributed computing environments.

Introduction

As distributed systems scale to include thousands of heterogeneous components—from data centers to individual computing units—understanding their "health" becomes a significant challenge. Every device and process generates logs, but the sheer volume and variety make manual inspection impossible. The challenge isn't just collecting data; it’s filtering out the noise. Maintainers need to know if a system failure is imminent, if a security breach is occurring, or if performance is degrading. This research addresses the lack of a standardized, end-to-end workflow for processing these logs in a way that provides a high-level "diagnosis" of the entire environment.

The Four-Stage Monitoring Framework

The proposed framework consists of four major stages designed to transform raw log data into actionable insights. By standardizing these steps, the framework moves away from ad-hoc log viewing toward a systematic diagnostic approach.

The general workflow for log processing, from selection to feedback.
  1. Log Selection: Unlike many approaches that try to ingest everything, this framework emphasizes human-driven selection. Maintainers identify which logs (e.g., syslog vs. middleware traces) are actually relevant to the specific goal, such as security or performance. This reduces computational burden and improves the signal-to-noise ratio.
  2. Log Classification: To handle millions of lines of text, the system categorizes logs into "types." This involves non-variable classification (identifying static templates) and variable classification (extracting specific IDs like IPs or process IDs).
  3. Information Analysis: This stage serves as the "brain" of the operation, applying statistical and machine learning models to the classified data.
  4. Result Feedback: The final stage converts complex mathematical outputs into human-readable formats, such as hierarchical timelines, 3D cube views, and heatmaps.

Information Analysis and System Diagnosis

The core of the framework lies in its ability to analyze different facets of system behavior through specialized algorithms. The ultimate goal is to move away from simple "log viewing" toward "system diagnosing," where various analysis results are integrated into a single "vulnerability level" or health score for the entire environment.

Hierarchy of log classification algorithms and their role in generating environment diagnosis reports.

The paper highlights three specific analysis applications:

  • Log Flow Anomaly Detection: Using Principal Component Analysis (PCA) to build a model of "normal" log volume. If the number of logs for a specific type spikes or drops unexpectedly, the system flags an anomaly.
  • Log Type Association: Mining event chains to determine if specific log types consistently precede others, identifying potential causal links in system failures.
  • User Behavior Modeling: Utilizing Deep Learning to build profiles of typical user interactions, enabling the detection of hijacked accounts or malicious activity.

Case Study: CNGrid

The framework was validated using real-world data from CNGrid (the China National Grid), which consists of 19 high-performance computing (HPC) clusters. Data sources included Linux syslogs (cron, maillog, secure, messages) and SCE Traces from middleware handling task scheduling.

Key results from this implementation include:

  • Visualizing Hotspots: PCA-based log flow detection successfully visualized hours of activity across multiple hosts in a single heatmap, allowing maintainers to spot hotspots instantly.
  • High Prediction Accuracy: Deep learning models for user behavior achieved an accuracy of over 90% for shorter sessions.
  • Real-time Monitoring: By slicing logs into 5-minute intervals, the system provided a near-real-time feedback loop.

Limitations and Future Work

Despite its robustness, the framework faces challenges. Accuracy for user behavior prediction drops to 60%–70% for long, complex sessions. Additionally, some event chains produce false positives where "normal" events are flagged as precursors to anomalies. Future work focuses on refining these models and developing more sophisticated weighting factors for the final "Environment Diagnosis Report" to ensure an accurate overall health score.

Summary updated

Abstract

Distributed systems have kept scaling upward since this concept appears, and they soon evolve to environments that contain heterogeneous components playing different roles, making it difficult to understand how the large environment works or if any undesired matters happened from security point of view. Logs, produced by devices, sub-systems and running processes, are a very important source to help system maintainers to get relative security knowledge. But there are too many logs and too many kinds of logs to deal with, which makes manual checking impossible. In this work we will share some of our experiences in log processing and analyzing. We have summarized some common major steps that appear in most of the existing log analysis approaches, including log selection, log classification, information analyses and result feedback. We also represent a general framework that monitors events, analyzes hidden information and diagnoses the healthy state for large distributed computing environments bases on logs. Although the framework we initially designed was for the maintenance for CNGrid, its process is adaptable to other distributed computing environments.
BibTeX
@misc{zhao2019blueprint,
  author = {Yining Zhao and Xiaodong Wang and Haili Xiao and Xuebin Chi},
  title = {{A Blueprint of Log Based Monitoring and Diagnosing Framework in Large Distributed Environments}},
  year = {2019},
  doi = {10.22323/1.351.0033},
}
Figure