Dynamic Adaptation of Temporal Event Correlation Rules

Rean Griffith, Gail E. Kaiser, Joseph M. Hellerstein, Yixin DiaoColumbia 2005 · DOI

TL;DR

This paper tackles the challenge of accurately detecting system failures and attacks in distributed systems by dynamically adjusting the timing parameters used in event correlation rules. They introduce an architecture that compensates for unpredictable network and server delays, achieving results comparable to or better than static, theoretically optimal settings.

What Problem Does This Paper Address?

Self-managing distributed systems rely heavily on temporal event correlation to detect anomalies like denial-of-service attacks or application issues. This involves correlating events that occur within specific time windows defined by "rules." The core problem is that these rules often use fixed timer values, which are brittle. These timers are susceptible to "propagation skew" – variations in the time it takes for events to travel from their source to the management station. This skew can be caused by network congestion, server load, or even unreliable clocks at event sources. If timers are too short, legitimate events might be missed (false negatives). If they are too long, minor fluctuations might trigger false alarms (false positives), overwhelming operators. The paper aims to address this by dynamically adapting these timer values to account for real-time propagation delays.

Methods

The authors propose an architecture that extends a conventional Management Station. Key components include:

  • Calibration Probes: These are special instrumentation points that generate known pairs of events (calibration frames) with predefined generation times.
  • Probe Monitor: This component measures the actual arrival times of these calibration events at the Management Station and calculates the propagation skew.
  • Controller: An autonomic control algorithm (detailed in a separate technical report [7]) uses the measured propagation skew to calculate a "slack time."
  • Management Station: The event correlation engine in the Management Station uses this calculated slack time to dynamically adjust the timer values of the temporal event patterns. Essentially, the slack time is added to the original timer value, effectively widening the window to account for delays.

The procedure for matching a temporal rule involves two events, E1 and E2. The system estimates the propagation skew (τ) by comparing the difference in arrival times at the Management Station with the expected generation time. The controller then adjusts the timer value (Ttmr) by adding a calculated slack time to compensate for this skew, aiming to reduce false alarms and missed detections.

Datasets

Not discussed in the paper. The paper describes experimental studies on a testbed system they built.

Key Insights and Ideas

  • Propagation Skew is a Significant Problem: The paper empirically demonstrates that propagation skews can be as large as 50% of the timer values used in sample rules, highlighting the necessity of addressing this variability.
  • Dynamic Adaptation is Feasible and Effective: The proposed architecture and adaptive control algorithm show that dynamically adjusting timer values based on measured propagation skew can significantly improve the accuracy of event correlation.
  • Self-Managing Approach: The adaptive control algorithm is designed to be self-managing, meaning it doesn't require manual parameterization or training, making it practical for real-world deployment.
  • Collocation Impacts Skew: Experiments revealed that collocated components (like the event router and event distiller on the same machine) can introduce more variable propagation skews, especially on Windows XP, due to resource contention.

Key Results

  • The experimental studies on their testbed system suggest that their adaptive approach produces results at least as good as, and in some cases better than, optimal fixed settings of timer values.
  • For a 2-machine configuration (Configuration C), where propagation skews were most variable, the adaptive algorithm achieved near-optimal results by selecting slack times very close to the value that maximized the Minimum Average Probability of a Correct result (MAPC).
  • In configurations A, B, and D, the adaptive algorithm also selected slack times very close to the optimal fixed slack time, demonstrating its general effectiveness.

Limitations and Future Work

  • Focus on Propagation Skew: This paper primarily focuses on compensating for propagation skew due to network and server delays. The authors mention that future work will address the case where event source clocks are globally synchronized but the timer values themselves are heuristic and uncertain.
  • Reinforcement Learning Not Discussed: While the authors suggest that reinforcement learning could be used to further optimize timer values by minimizing a weighted function of false and undetected alarms, this approach is not detailed in this paper.
  • Testbed Specifics: The experiments were conducted on a specific testbed with particular configurations. Generalizing these results to vastly different system architectures and workloads would require further validation.

Summary updated

Abstract

Temporal event correlation is essential to realizing self-managing distributed systems. Autonomic controllers often require that events be correlated across multiple components using rule patterns with timer-based transitions, e.g., to detect denial of service attacks and to warn of staging problems with business critical applications. This short paper discusses automatic adjustment of timer values for event correlation rules, in particular compensating for the variability of event propagation delays due to factors such as contention for network and server resources. We describe a corresponding Management Station architecture and present experimental studies on a testbed system that suggest that this approach can produce results at least as good as an optimal fixed setting of timer values.
BibTeX
@inproceedings{griffith2005dynamic,
  author = {Rean Griffith and Gail E. Kaiser and Joseph M. Hellerstein and Yixin Diao},
  title = {{Dynamic Adaptation of Temporal Event Correlation Rules}},
  booktitle = {Columbia},
  year = {2005},
  doi = {10.7916/d8j96cpw},
}

Loading related content...

Figure