DEPEND: A simulation-based environment for system level dependability analysis

Kumar K. Goswami, Ravishankar K. IyerIllinois 1992

Abstract

The design and evaluation of highly reliable computer systems is a complex issue. Designers mostly develop such systems based on prior knowledge and experience and occasionally from analytical evaluations of simplified designs. A simulation-based environment called DEPEND, especially geared for the design and evaluation of fault-tolerant architectures, is presented. DEPEND exploits the properties of object-oriented programming to provide a flexible framework with which a user can rapidly model and evaluate various fault-tolerant systems. This paper describes the key features of the DEPEND environment and illustrates its capabilities with a detailed analysis of a real design: the Unix-based Tandem Integrity fault-tolerance system.

Introduction: The Design Gap in Fault-Tolerant Systems

Designing highly reliable systems presents a significant challenge. Engineers often face a "catch-22": analytical models, such as Markov chains, are computationally efficient but frequently too simplified to capture complex real-world behaviors like error propagation or latent faults. Conversely, physical fault injection—testing a physical prototype by inducing failures—is highly accurate but can only be performed late in the development cycle, often after the core architecture is finalized.

The DEPEND environment addresses this gap by providing a software-based platform to "build" and "break" systems during the design phase. A primary focus of the tool is the analysis of near-coincident errors: scenarios where a second fault occurs before the system has recovered from the first. These events are a leading cause of failure in redundant, fault-tolerant architectures.

The DEPEND Environment

DEPEND is built using an object-oriented approach in C++, providing a library of objects that represent standard system components such as CPUs, communication links, voters, and RAM. This modularity allows for rapid prototyping of complex architectures.

Overview of the DEPEND simulation environment architecture.

The environment utilizes several key techniques to ensure both flexibility and performance:

  • Functional Fault Modeling: Rather than simulating low-level gate or transistor behavior, DEPEND models the functional impact of faults (e.g., a processor hang or message corruption). This high-level abstraction enables the simulation of entire systems.
  • Time Acceleration: To simulate rare failure events that might occur only once every few years, DEPEND employs an "error-driven" approach. The simulator leaps forward to the next significant event—such as a fault arrival or a latent error activation—skipping idle periods and providing massive speedups.
  • Hybrid Simulation: The environment can execute actual software code within the simulation, allowing researchers to observe how specific algorithms, such as load balancing or recovery logic, interact with hardware failures.

Validation and Case Study: The Tandem Integrity

To validate the accuracy of DEPEND, the authors modeled the Tandem Integrity, a production fault-tolerant machine. They compared simulation results against data obtained from physical fault-injection experiments using a Tektronix DAS 9200 logic analyzer.

The simulation parameters were derived from empirical studies of existing systems (like DEC VAXclusters). The results demonstrated that the distribution of "Time to CPU Shutdown" generated by DEPEND was statistically identical to the physical measurements, confirming that the high-level simulation could accurately predict complex system behavior under stress.

Key Insights and Design Implications

The study yielded several counter-intuitive insights into system reliability:

  • The Latency Paradox: Error latency—the time a fault remains dormant before detection—can actually improve reliability in the presence of correlated errors. By staggering detection times, latency reduces the likelihood that two errors are detected simultaneously, which would otherwise lead to a total system crash.
  • The Reliability Bottleneck: In the Tandem Integrity case study, the primary "window of vulnerability" was found to be the global memory re-integration time rather than the CPU repair time. Under heavy workloads, this re-integration period extends, significantly decreasing the Mean Time Between Failures (MTBF).
  • Scrubbing Trade-offs: While memory scrubbing is essential for mitigating small error latencies, it offers diminishing returns when dealing with highly correlated faults.

Conclusion

DEPEND demonstrates that object-oriented simulation can bridge the gap between abstract analytical models and late-stage physical testing. By providing a validated, time-accelerated environment, it allows designers to optimize parameters—such as reducing CPU Power-On Self-Test (POST) times—to significantly improve system MTBF before a single piece of hardware is built. Future work aims to integrate these simulation-derived latency distributions into even more efficient analytical models.

Summary updated

Abstract

The design and evaluation of highly reliable computer systems is a complex issue. Designers mostly develop such systems based on prior knowledge and experience and occasionally from analytical evaluations of simplified designs. A simulation-based environment called DEPEND which is especially geared for the design and evaluation of fault-tolerant architectures is presented. DEPEND is unique in that it exploits the properties of object-oriented programming to provide a flexible framework with which a user can rapidly model and evaluate various fault-tolerant systems. The key features of the DEPEND environment are described, and its capabilities are illustrated with a detailed analysis of a real design. In particular, DEPEND is used to simulate the Unix based Tandem Integrity fault-tolerance and evaluate how well it handles near-coincident errors caused by correlated and latent faults. Issues such as memory scrubbing, re-integration policies, and workload dependent repair times which affect how the system handles near-coincident errors are also evaluated. Issues such as the method used by DEPEND to simulate error latency and the time acceleration technique that provides enormous simulation speed up are also discussed. Unlike any other simulation-based dependability studies, the use of these approaches and the accuracy of the simulation model are validated by comparing the results of the simulations, with measurements obtained from fault injection experiments conducted on a production Tandem Integrity machine.
BibTeX
@inproceedings{goswami1992depend,
  author = {Kumar K. Goswami and Ravishankar K. Iyer},
  title = {{DEPEND: A simulation-based environment for system level dependability analysis}},
  booktitle = {Illinois},
  year = {1992},
}
Figure