DEPEND: A design environment for prediction and evaluation of system dependability

Kumar K. Goswami, Ravishankar K. Iyer

Abstract

As computer systems moved into mission-critical roles in the late 1980s, the need for rigorous dependability analysis became paramount. This paper presents DEPEND, an integrated simulation environment developed at the University of Illinois. DEPEND provides a functional-level modeling framework that automates fault injection and performance monitoring. Its primary innovation is a workload-dependent fault injector that targets systems during peak stress periods, revealing vulnerabilities that traditional random testing might overlook.

Introduction

By 1990, the complexity of fault-tolerant distributed systems had outpaced the capabilities of traditional analytical tools. Designers faced a significant challenge: how to accurately predict how a system would behave when hardware or software components failed under load. Manual fault injection was labor-intensive and often failed to capture the subtle interactions between intermittent errors and system workload. DEPEND was created to bridge this gap, offering an automated, object-oriented environment for stress-testing system designs before physical implementation.

The DEPEND Architecture

DEPEND is built as an object-oriented wrapper around CSIM, a process-based simulation language. The architecture is modular, allowing for the simulation of various hardware and software components at a functional level.

The hierarchical architecture of the DEPEND simulation environment, showing the integration of the PARAGRAPH visualization tool, the simulation model, and the underlying CSIM engine.

As shown in Figure 1, the environment consists of several integrated layers:

  • Simulation Model: Written in C++, this layer utilizes the DEPEND object library to model components like CPUs, disks, and communication channels.
  • Fault Injector: A dedicated module that automates the introduction of failures based on predefined strategies, including random and workload-dependent models.
  • PARAGRAPH: A graphical tool that provides both on-line and off-line viewing of performance indicators, allowing researchers to visualize system degradation in real-time.
  • CSIM Engine: The underlying discrete-event simulation core that handles process scheduling and resource management.

Methodology: Workload-Dependent Fault Injection

The core technical contribution of DEPEND is its approach to fault injection. While it supports standard random (exponential) failure timing, its standout feature is workload-dependent injection.

Real-world data suggests that systems are more prone to failure when pushed to their limits. DEPEND utilizes a statistical clustering algorithm to monitor the system's workload states. When the simulation detects a "high-stress" state—characterized by high CPU or network utilization—the probability of injecting a fault increases. This allows designers to observe how "intermittent" errors, which may appear and disappear rapidly, can cause a system to enter a performance "death spiral."

Experimental Evaluation and Results

To validate the environment, the authors simulated a 10-processor distributed system using actual workload trace files from a VAX-11/780. The study focused on a dynamic load-balancing heuristic known as MINQ.

The "Node 0" Bottleneck

The simulation revealed a critical design flaw: when the communication channel experienced message loss, the central scheduler (Node 0) became overwhelmed. This was not a mathematical error in the MINQ algorithm itself, but a subtle implementation detail regarding how the scheduler handled status updates.

Performance Impact

The results highlighted the severe impact of intermittent failures:

  • Response Time Spikes: Persistent intermittent failures lasting just two minutes resulted in a 46% increase in average process response time.
  • Catastrophic Message Loss: Losing 10% of status update messages led to a staggering 300% increase in response time.
  • The Fix: Using insights from DEPEND, the authors modified the architecture so the central scheduler shared communication risks more equitably. This adjustment reduced the performance hit from 300% down to a manageable 16% under identical failure conditions.

Conclusion

DEPEND demonstrated that functional-level simulation is a powerful tool for identifying architectural bottlenecks early in the design process. By correlating failures with workload, the environment provides a more realistic assessment of system dependability than traditional random models. While limited to functional behaviors rather than low-level hardware physics, DEPEND offers a scalable and automated path toward building more resilient mission-critical systems.

Summary updated

Abstract

Continu on reverse If necessery and identify by block numbr) fa u lt-to le ra n c e , design, e valuation, sim ulation, fa u lt -in je c t io n , d istrib u te d systems 19.ABSTRACT (Continue on reverse if necessery and identify by block num ber)This paper describes the development of DEPEND, an integrated simulation environment for the design and dependability analysis of fault-tolerant systems.DEPEND models both hardware and software components at a functional level, and allows automatic failure injection to assess system performance and re liability.It relieves the user of the work needed to inject failures, maintain statistics and output reports.The automatic failure injection scheme is geared toward evaluating a system under high stress (workload) conditions.The fail ures which are injected can affect both hardware and software components.To illustrate the capability of the simulator, a distributed system which employs a prediction-based, dynamic load-balancing heuristic is evaluated.Experiments are conducted to determine the impact of failures on system performance and, to identify the failures to which the system is especially susceptible.
BibTeX
@inproceedings{goswami1990depend,
  author = {Kumar K. Goswami and Ravishankar K. Iyer},
  title = {{DEPEND: A design environment for prediction and evaluation of system dependability}},
  booktitle = {Illinois},
  year = {1990},
}
Figure