TL;DR
This paper introduces a methodology for rigorously evaluating self-healing systems by combining fault injection with mathematical modeling (specifically, Continuous Time Markov Chains - CTMCs). The authors demonstrate how this approach can quantify the impact of faults and the effectiveness of various self-healing mechanisms, providing a crucial step towards building objective benchmarks for these complex systems.
What Problem Does This Paper Address?
Evaluating the effectiveness of self-healing systems is a significant challenge. These systems aim to automatically detect, diagnose, and repair software and hardware problems. However, creating a robust benchmark for them is difficult due to several factors:
- Quantifying Fault Impact: It's hard to establish a baseline to measure improvements.
- Diverse Healing Styles: Systems can heal reactively, preventatively, or proactively, and these need to be accounted for.
- Partial and Imperfect Healing: Repairs aren't always complete or successful, and some healing might be automated while other parts require manual intervention.
Without a standardized, objective, and quantitative way to evaluate these systems, it's difficult to compare them, understand their true benefits, or even reason about their expected performance in real-world scenarios.
Methods
The core of the proposed methodology lies in the combination of two key components:
- Practical Fault-Injection Tools:
- Kheiron/JVM: This tool uses bytecode rewriting to inject faults into Java applications, simulating issues like memory leaks, hangs, and delays.
- Nooks Device-Driver Fault-Injection Tools: This leverages the Linux kernel module interface to inject faults into device drivers, including text faults, memory leaks, and hangs.
- Mathematical Modeling Techniques:
- Continuous Time Markov Chains (CTMCs): These are used to model the system's states (e.g., UP, DOWN, RECOVERING) and transitions between them, allowing for the calculation of metrics like steady-state availability, downtime, repair success rates, and repair times.
- Markov Reward Networks: These extend CTMCs to incorporate costs associated with downtime, such as financial losses or SLA penalties.
The paper proposes a "7U-Evaluation Method" which involves: gathering fault data, establishing fault-remediation relationships, selecting/creating fault-injection tools, identifying macro-measurements (system-level constraints like SLAs) and micro-measurements (specific healing mechanism metrics), running fault-injection experiments, and constructing pre- and post-experiment models.
Datasets
The paper does not explicitly mention using pre-existing datasets for training or evaluation. Instead, it focuses on creating a methodology that can be applied to various systems. The experiments described use a "TPC-W Web-application" running on a 3-tier architecture (Webserver, App server, DB server) with specific software versions (Resin 3.0.22, Sun Hotspot JVM v1.5, MySQL 5.0.27, Linux 2.4.18). The "Real-World Downtime Data" section cites external reports from "StackSafe" and "Research Edge, Inc." for context on typical downtime incidents and costs.
Key Insights and Ideas
- RAS as a Universal Metric: Reliability, Availability, and Serviceability (RAS) are presented as effective and reasonable metrics for evaluating self-healing systems, bridging the gap between engineering and business concerns.
- CTMCs + Fault Injection = Powerful Evaluation: Combining practical fault injection with analytical CTMC modeling provides a feasible and flexible framework for evaluating and comparing the RAS characteristics of computing systems. This synergy allows for both realistic fault simulation and quantitative analysis.
- The "Easy" Part is the Math: The authors highlight that while mathematical modeling is crucial, the real challenge lies in obtaining accurate, real-world data on failures, outages, and their impacts. This data is critical for defining what "better" and "good" truly mean for self-healing systems.
- Understanding Trade-offs: The methodology allows for quantifying the impact of specific healing mechanism details (like success rates and recovery times) on operational constraints (SLAs, availability), enabling informed decisions about system design and optimization.
Key Results
The paper presents three examples demonstrating the application of their methodology:
- Example 1: Resin App Server: Analyzing perfect recovery of the application server showed a steady-state availability of 99.838%, resulting in 866 minutes of downtime per year. This analysis also provided expected penalties for different availability guarantees.
- Example 2: Linux w/Nooks: This example focused on imperfect recovery of device drivers using Nooks. The analysis showed how system availability varies with the Nooks remediation success rate, illustrating that even with imperfect healing, significant availability improvements are possible.
- Example 3: Resin + Linux + Nooks: By composing Markov chains from the previous examples, the authors modeled a complete system. This composite model achieved a maximum availability of 99.835% with a minimum downtime of 866 minutes, demonstrating the ability to analyze complex interactions between different self-healing components.
Limitations and Future Work
Limitations:
- CTMC Assumptions: CTMCs rely on the assumption of constant hazard rates, which may not always hold true for real-world fault distributions.
- Fault Independence: The models often assume fault independence, which can be limiting when analyzing near-coincident or cascading faults.
- Replicability of Failures: Some failures are inherently harder to replicate or induce in a controlled environment, impacting the accuracy of fault-injection experiments.
- Data Availability: As mentioned, obtaining detailed, real-world failure data remains a significant challenge.
Future Work:
- Expanded Experiments: Conduct more experiments on a wider range of operating systems (Linux 2.6, OpenSolaris 10, Windows XP SP2/2003 Server) and server applications.
- Analyze Other Mechanisms: Model and analyze different self-healing mechanisms, such as Error Virtualization and self-healing in OpenSolaris.
- Feedback Control: Develop feedback control mechanisms for policy-driven repair strategy selection.
Summary updated