Abstract
Hardware transient faults, such as bit flips, pose a significant threat to the reliability of Deep Neural Networks (DNNs) in safety-critical applications. This paper introduces PyTorchALFI, an open-source extension to the PyTorchFI framework designed for large-scale, repeatable hardware fault injection (FI) campaigns. PyTorchALFI addresses the scalability and reproducibility challenges of existing software-level FI tools by utilizing configuration-driven testing and pre-generated fault matrices. We demonstrate the framework's utility by evaluating the resilience of various classification and object detection architectures and assessing the effectiveness of several protection strategies.
Introduction
As semiconductor technology advances, transistors shrink and operating voltages decrease, making hardware increasingly susceptible to transient faults like bit flips. In the context of deep neural networks (DNNs), these faults can manifest as Silent Data Errors (SDEs)—instances where a model produces incorrect outputs without triggering system alerts. For safety-critical domains like autonomous driving, quantifying and mitigating SDEs is paramount. However, existing hardware-level simulations are prohibitively slow for modern DNNs, while current software-level tools often lack the scalability and reproducibility required for rigorous safety validation.
PyTorchALFI Framework
PyTorchALFI builds upon the PyTorchFI framework, leveraging PyTorch "hooks" to simulate faults by modifying neuron values during inference and performing direct tensor manipulation for weight-level faults.
Architectural Innovations
The framework introduces several key components to streamline large-scale testing:
- Alficore Component: Acts as a central integration layer, managing data loaders, monitoring for anomalies (such as NaNs or Infs), and coordinating evaluation metrics.
- Configuration-Driven Testing: Users define fault parameters—including total faults, types, locations, and bit positions—via YAML configuration files, simplifying experiment setup.
- Fault Matrices: To ensure strict reproducibility, all faults are pre-generated into a matrix and stored as binary files. This allows researchers to apply the exact same fault scenarios across different model iterations or protection schemes.
- Layer-Specific Targeting: The tool supports weighted random selection of layers, allowing for focused analysis on the most vulnerable parts of a network.
Experimental Evaluation
The framework was evaluated using standard computer vision datasets, including COCO for classification and detection, and KITTI and Lyft for specialized object detection tasks.
Key Insights
- Reproducibility: Decoupling fault generation from execution enables "apples-to-apples" comparisons between baseline and hardened models.
- Bit Sensitivity: PyTorchALFI allows targeting specific bit positions, revealing that flipping exponent bits in floating-point representations is significantly more disruptive than flipping mantissa bits.
- Architectural Variance: Different DNN architectures exhibit varying degrees of sensitivity to faults, necessitating model-specific vulnerability assessments.
Results and Discussion
Our analysis revealed significant vulnerabilities in unprotected models. For image classification, VGG-16 exhibited an 11.8% SDE rate under a single fault per image, while ResNet-50 and AlexNet showed rates of 8.7% and 10.9%, respectively. In object detection tasks, RetinaNet trained on COCO reached a 4.2% rate of incorrect detections.
The framework also proved effective in validating mitigation strategies. As shown in the comparative analysis, protection methods such as "Clipper" and "BackFlip" can reduce SDE rates to near zero across multiple architectures.
Conclusion
PyTorchALFI provides a robust, open-source solution for large-scale fault injection, bridging the gap between slow hardware simulations and non-reproducible software tools. By enabling efficient SDE quantification and mitigation validation, it supports the development of safety cases for AI in critical applications. Future work will focus on integrating more realistic hardware fault models and expanding support for a broader range of model architectures.
The framework is available as open-source code on GitHub.
Summary updated