FTT-NAS: Discovering Fault-tolerant Convolutional Neural Architecture

Xuefei Ning, Guangjun Ge, Wenshuo Li, Zhenhua Zhu, Yin Zheng, Xiaoming Chen, Zhen Gao, Yu Wang, Huazhong YangACM Trans. Design Autom. Electr. Syst 2021 · DOI · OpenReview

Introduction

As deep learning transitions from cloud data centers to edge devices like autonomous vehicles, satellites, and IoT sensors, it faces a harsh reality: hardware is imperfect. In these environments, neural networks (NNs) are susceptible to soft errors (transient bit-flips from radiation or voltage drops) and hard errors (permanent defects like Stuck-at-Faults in RRAM). Traditionally, engineers have relied on hardware redundancy, such as Triple Modular Redundancy (TMR), to mitigate these risks. However, TMR can double or triple power and area costs, which is often prohibitive for resource-constrained edge devices.

FTT-NAS (Fault-Tolerant Neural Architecture Search) proposes a different approach: designing the neural network's architecture itself to be naturally resilient to hardware faults.

Hardware Fault Modeling

To search for resilient architectures, the authors first formalized how hardware faults manifest at the application level. They focused on two primary models:

  1. MiBB (MAC-i.i.d Bit-Bias): This models feature-map faults in FPGAs and ASICs, where errors accumulate within the adder trees of Multiply-Accumulate (MAC) units.
  2. adSAF (Arbitrary-distributed Stuck-at-Fault): This models weight faults in emerging memory technologies like Resistive RAM (RRAM), where cells become permanently stuck at specific values.
Hardware components and their associated error models (iBF and iBB) in FPGA and ASIC platforms.

As shown in the hardware landscape above, different components exhibit different error characteristics. While some parts like control logic are difficult to model algorithmically, the computational logic and buffers can be effectively simulated using statistical bit-flip and bit-bias models.

The FTT-NAS Framework

The core of FTT-NAS is an automated search process that optimizes for both "clean" accuracy and "faulty" accuracy. The search objective is defined by a multi-objective reward: $Reward = (1 - \alpha) \cdot acc_{clean} + \alpha \cdot acc_{faulty}$.

A critical component of this framework is Fault-Tolerant Training (FTT). During the search, random faults are injected into the "super-net"—the large search space from which specific architectures are sampled. This ensures that the weights learned during the search process are inherently adapted to noisy hardware environments.

The Fragility of Standard Architectures

The motivation for FTT-NAS is underscored by the extreme fragility of popular manually designed architectures. When subjected to even low rates of hardware faults, the performance of models like MobileNet-V2 degrades rapidly.

Accuracy of standard CNN models under varying feature-map fault rates.

As seen in the table, while ResNet-18 maintains some level of functionality at a $10^{-5}$ fault rate, MobileNet-V2 collapses to near-random guessing (10%) at a $10^{-4}$ fault rate. This highlights that efficiency-focused designs often sacrifice robustness.

Key Insights and Results

The search process revealed several architectural "rules of thumb" for fault tolerance:

  • The "Fewer MACs" Rule: For feature-map faults, the controller preferred operations like Separable and Dilated Convolutions. By reducing the number of MAC operations per output pixel, these layers provide fewer opportunities for bit-flips to occur in the hardware adder tree.
  • The "Small Weights" Rule: For RRAM weight faults, the search favored ReLU-Conv-BN blocks. These structures tend to produce smaller weight magnitudes, which minimizes the impact of a "stuck-at-maximum" fault.

The resulting models, F-FTT-Net and W-FTT-Net, demonstrated significant gains. Under a high feature fault rate ($3 \times 10^{-4}$), F-FTT-Net achieved 86.2% accuracy on CIFAR-10, whereas MobileNet-V2 failed completely. Similarly, under an 8% weight fault rate in RRAM, W-FTT-Net reached 69.6% accuracy, significantly outperforming ResNet-18 (60.8%).

Conclusion and Future Work

FTT-NAS demonstrates that architectural design is a powerful lever for hardware reliability. By moving beyond simple redundancy and into the realm of algorithmic resilience, we can deploy robust AI in challenging edge environments without the typical overhead of TMR. Future research will look toward scaling these findings to larger datasets like ImageNet and addressing the more complex challenge of faults in hardware control logic.

Summary updated

Abstract

With the fast evolvement of embedded deep-learning computing systems, applications powered by deep learning are moving from the cloud to the edge. When deploying neural networks (NNs) onto the devices under complex environments, there are various types of possible faults: soft errors caused by cosmic radiation and radioactive impurities, voltage instability, aging, temperature variations, malicious attackers, and so on. Thus, the safety risk of deploying NNs is now drawing much attention. In this article, after the analysis of the possible faults in various types of NN accelerators, we formalize and implement various fault models from the algorithmic perspective. We propose Fault-Tolerant Neural Architecture Search (FT-NAS) to automatically discover convolutional neural network (CNN) architectures that are reliable to various faults in nowadays devices. Then, we incorporate fault-tolerant training (FTT) in the search process to achieve better results, which is referred to as FTT-NAS. Experiments on CIFAR-10 show that the discovered architectures outperform other manually designed baseline architectures significantly, with comparable or fewer floating-point operations (FLOPs) and parameters. Specifically, with the same fault settings, F-FTT-Net discovered under the feature fault model achieves an accuracy of 86.2% (VS. 68.1% achieved by MobileNet-V2), and W-FTT-Net discovered under the weight fault model achieves an accuracy of 69.6% (VS. 60.8% achieved by ResNet-18). By inspecting the discovered architectures, we find that the operation primitives, the weight quantization range, the capacity of the model, and the connection pattern have influences on the fault resilience capability of NN models.
BibTeX
@article{ning2021ftt,
  author = {Xuefei Ning and Guangjun Ge and Wenshuo Li and Zhenhua Zhu and Yin Zheng and Xiaoming Chen and Zhen Gao and Yu Wang and Huazhong Yang},
  title = {{FTT-NAS: Discovering Fault-tolerant Convolutional Neural Architecture}},
  journal = {ACM Trans. Design Autom. Electr. Syst.},
  year = {2021},
  doi = {10.1145/3460288},
}
Figure