Abstract
The design and evaluation of highly reliable computer systems is a complex issue. Designers mostly develop such systems based on prior knowledge and experience and occasionally from analytical evaluations of simplified designs. A simulation-based environment called DEPEND, especially geared for the design and evaluation of fault-tolerant architectures, is presented. DEPEND exploits the properties of object-oriented programming to provide a flexible framework with which a user can rapidly model and evaluate various fault-tolerant systems. This paper describes the key features of the DEPEND environment and illustrates its capabilities with a detailed analysis of a real design: the Unix-based Tandem Integrity fault-tolerance system.
Introduction: The Design Gap in Fault-Tolerant Systems
Designing highly reliable systems presents a significant challenge. Engineers often face a "catch-22": analytical models, such as Markov chains, are computationally efficient but frequently too simplified to capture complex real-world behaviors like error propagation or latent faults. Conversely, physical fault injection—testing a physical prototype by inducing failures—is highly accurate but can only be performed late in the development cycle, often after the core architecture is finalized.
The DEPEND environment addresses this gap by providing a software-based platform to "build" and "break" systems during the design phase. A primary focus of the tool is the analysis of near-coincident errors: scenarios where a second fault occurs before the system has recovered from the first. These events are a leading cause of failure in redundant, fault-tolerant architectures.
The DEPEND Environment
DEPEND is built using an object-oriented approach in C++, providing a library of objects that represent standard system components such as CPUs, communication links, voters, and RAM. This modularity allows for rapid prototyping of complex architectures.
The environment utilizes several key techniques to ensure both flexibility and performance:
- Functional Fault Modeling: Rather than simulating low-level gate or transistor behavior, DEPEND models the functional impact of faults (e.g., a processor hang or message corruption). This high-level abstraction enables the simulation of entire systems.
- Time Acceleration: To simulate rare failure events that might occur only once every few years, DEPEND employs an "error-driven" approach. The simulator leaps forward to the next significant event—such as a fault arrival or a latent error activation—skipping idle periods and providing massive speedups.
- Hybrid Simulation: The environment can execute actual software code within the simulation, allowing researchers to observe how specific algorithms, such as load balancing or recovery logic, interact with hardware failures.
Validation and Case Study: The Tandem Integrity
To validate the accuracy of DEPEND, the authors modeled the Tandem Integrity, a production fault-tolerant machine. They compared simulation results against data obtained from physical fault-injection experiments using a Tektronix DAS 9200 logic analyzer.
The simulation parameters were derived from empirical studies of existing systems (like DEC VAXclusters). The results demonstrated that the distribution of "Time to CPU Shutdown" generated by DEPEND was statistically identical to the physical measurements, confirming that the high-level simulation could accurately predict complex system behavior under stress.
Key Insights and Design Implications
The study yielded several counter-intuitive insights into system reliability:
- The Latency Paradox: Error latency—the time a fault remains dormant before detection—can actually improve reliability in the presence of correlated errors. By staggering detection times, latency reduces the likelihood that two errors are detected simultaneously, which would otherwise lead to a total system crash.
- The Reliability Bottleneck: In the Tandem Integrity case study, the primary "window of vulnerability" was found to be the global memory re-integration time rather than the CPU repair time. Under heavy workloads, this re-integration period extends, significantly decreasing the Mean Time Between Failures (MTBF).
- Scrubbing Trade-offs: While memory scrubbing is essential for mitigating small error latencies, it offers diminishing returns when dealing with highly correlated faults.
Conclusion
DEPEND demonstrates that object-oriented simulation can bridge the gap between abstract analytical models and late-stage physical testing. By providing a validated, time-accelerated environment, it allows designers to optimize parameters—such as reducing CPU Power-On Self-Test (POST) times—to significantly improve system MTBF before a single piece of hardware is built. Future work aims to integrate these simulation-derived latency distributions into even more efficient analytical models.
Summary updated