AEDAM: Whole Program Adaptive Error Detection and Mitigation (Final Report)

Mattan Erez, Sriram Krishnamoorthy, Gökçen Kestor, Joseph Manzano, Leon Song, Omer Subasi, Burcu O. Mutlu, Saman Amarasinghe, Michael Carbin, Martin Rinard, P. Sadayappan, Gagan Agrawal, Scott Mahlke, Ganesh Gopalakrishnan2021 · DOI

Abstract

The AEDAM project aims to transform software transient-error detection through configurable specialized detectors and quantitative characterization of hardware resilience. UT Austin's contributions focus on the Hamartia open-source error injection framework and the Nested Monte Carlo (NMC) methodology. These tools enable the study of detector composition, allowing researchers to efficiently protect whole programs against transient hardware faults.

Introduction: The Reliability Challenge at Exascale

As computing moves toward the exascale, "soft errors"—transient hardware faults such as bit-flips caused by cosmic rays or voltage fluctuations—pose a significant threat to system reliability. To mitigate these risks, researchers employ error injection to evaluate how systems fail and how various detectors (e.g., software assertions or hardware residue checkers) perform.

However, current tools face a critical "Fidelity vs. Speed" trade-off. Simple bit-flip models are fast but inaccurate, while high-fidelity Register-Transfer Level (RTL) simulations are often too slow for large-scale applications. Furthermore, the "Masking Problem" in traditional Monte Carlo simulations leads to thousands of wasted runs where injected errors have no impact on the final output.

Methodology: High-Fidelity Injection and Efficient Sampling

The Hamartia Framework

The UT Austin team developed Hamartia, a fully open-source suite that integrates high-fidelity hardware error models with Intel’s Pin dynamic binary instrumentation platform. Hamartia utilizes on-demand, just-in-time logic simulation of arithmetic unit RTL models. This approach allows the framework to simulate how a specific circuit fails based on real-time input values, bridging the gap between speed and fidelity.

Nested Monte Carlo (NMC)

To address the masking problem, the researchers introduced the Nested Monte Carlo (NMC) methodology. Instead of executing a full application for every injection, NMC utilizes "micro-rollbacks." If an error is immediately masked or caught by a local detector, the framework re-executes only the affected instruction or function until an unmasked, undetected error is produced.

Nested Monte Carlo (NMC) methodology flowchart compared to simple random sampling.

Key Insights and Results

Performance and Scalability

Hamartia demonstrates superior performance compared to PinFi, previously the fastest injection framework, while providing significantly higher modeling fidelity. A key advantage is its fixed overhead; regardless of application duration, Hamartia maintains a consistent slowdown (approximately 4x), ensuring scalability for long-running HPC applications.

The Necessity of High Fidelity

A major finding of the project is that while low-fidelity bit-flip models may suffice for general sensitivity analysis, they are inadequate for evaluating specific error detectors. Because detectors are often tailored to specific circuits and data values, high-fidelity RTL models are essential to accurately measure their effectiveness.

Synergy in Detector Composition

The study highlights that software and hardware detectors "compose" effectively. By combining simple software assertions with basic hardware detectors, researchers achieved protection levels equivalent to much more complex and computationally expensive standalone hardware solutions.

Error outcome ratios for different detector compositions in the CLAMR miniapp.

As shown in the CLAMR benchmarks, composing detectors (e.g., RTL+) significantly reduces the ratio of Silent Data Corruption (SDC) compared to baseline configurations, demonstrating the power of multi-layered defense.

Conclusion and Future Work

The AEDAM project has successfully delivered tools that reduce the simulation time required for error injection campaigns by 10x to 100x. Future work will focus on expanding support for floating-point hardware detectors and conducting broader composition studies in increasingly complex system scenarios.

Summary updated

Abstract

The overall goals of the AEDAM project were to fundamentally transform software transient-error detection through the design of configurable specialized detectors, quantitative characterization of hardware resilience and software vulnerabilities, and composition of the specialized detectors to most efficiently protect the whole program. Within this scope, UT Austin's research contribution related to enabling and studying the composition of detectors and possible different hardware errors, specifically: (1) developed the Hamartia open-source error injection framework that is designed to make composition studies simple, and (2) develop the methodology and demonstrate the potential benefits of error detector composition.
BibTeX
@misc{erez2021aedam,
  author = {Mattan Erez and Sriram Krishnamoorthy and Gökçen Kestor and Joseph Manzano and Leon Song and Omer Subasi and Burcu O. Mutlu and Saman Amarasinghe and Michael Carbin and Martin Rinard and P. Sadayappan and Gagan Agrawal and Scott Mahlke and Ganesh Gopalakrishnan},
  title = {{AEDAM: Whole Program Adaptive Error Detection and Mitigation (Final Report)}},
  year = {2021},
  doi = {10.2172/1642550},
}
Figure