Abstract
As computer systems moved into mission-critical roles in the late 1980s, the need for rigorous dependability analysis became paramount. This paper presents DEPEND, an integrated simulation environment developed at the University of Illinois. DEPEND provides a functional-level modeling framework that automates fault injection and performance monitoring. Its primary innovation is a workload-dependent fault injector that targets systems during peak stress periods, revealing vulnerabilities that traditional random testing might overlook.
Introduction
By 1990, the complexity of fault-tolerant distributed systems had outpaced the capabilities of traditional analytical tools. Designers faced a significant challenge: how to accurately predict how a system would behave when hardware or software components failed under load. Manual fault injection was labor-intensive and often failed to capture the subtle interactions between intermittent errors and system workload. DEPEND was created to bridge this gap, offering an automated, object-oriented environment for stress-testing system designs before physical implementation.
The DEPEND Architecture
DEPEND is built as an object-oriented wrapper around CSIM, a process-based simulation language. The architecture is modular, allowing for the simulation of various hardware and software components at a functional level.
As shown in Figure 1, the environment consists of several integrated layers:
- Simulation Model: Written in C++, this layer utilizes the DEPEND object library to model components like CPUs, disks, and communication channels.
- Fault Injector: A dedicated module that automates the introduction of failures based on predefined strategies, including random and workload-dependent models.
- PARAGRAPH: A graphical tool that provides both on-line and off-line viewing of performance indicators, allowing researchers to visualize system degradation in real-time.
- CSIM Engine: The underlying discrete-event simulation core that handles process scheduling and resource management.
Methodology: Workload-Dependent Fault Injection
The core technical contribution of DEPEND is its approach to fault injection. While it supports standard random (exponential) failure timing, its standout feature is workload-dependent injection.
Real-world data suggests that systems are more prone to failure when pushed to their limits. DEPEND utilizes a statistical clustering algorithm to monitor the system's workload states. When the simulation detects a "high-stress" state—characterized by high CPU or network utilization—the probability of injecting a fault increases. This allows designers to observe how "intermittent" errors, which may appear and disappear rapidly, can cause a system to enter a performance "death spiral."
Experimental Evaluation and Results
To validate the environment, the authors simulated a 10-processor distributed system using actual workload trace files from a VAX-11/780. The study focused on a dynamic load-balancing heuristic known as MINQ.
The "Node 0" Bottleneck
The simulation revealed a critical design flaw: when the communication channel experienced message loss, the central scheduler (Node 0) became overwhelmed. This was not a mathematical error in the MINQ algorithm itself, but a subtle implementation detail regarding how the scheduler handled status updates.
Performance Impact
The results highlighted the severe impact of intermittent failures:
- Response Time Spikes: Persistent intermittent failures lasting just two minutes resulted in a 46% increase in average process response time.
- Catastrophic Message Loss: Losing 10% of status update messages led to a staggering 300% increase in response time.
- The Fix: Using insights from DEPEND, the authors modified the architecture so the central scheduler shared communication risks more equitably. This adjustment reduced the performance hit from 300% down to a manageable 16% under identical failure conditions.
Conclusion
DEPEND demonstrated that functional-level simulation is a powerful tool for identifying architectural bottlenecks early in the design process. By correlating failures with workload, the environment provides a more realistic assessment of system dependability than traditional random models. While limited to functional behaviors rather than low-level hardware physics, DEPEND offers a scalable and automated path toward building more resilient mission-critical systems.
Summary updated