Introduction
As deep learning transitions from cloud data centers to edge devices like autonomous vehicles, satellites, and IoT sensors, it faces a harsh reality: hardware is imperfect. In these environments, neural networks (NNs) are susceptible to soft errors (transient bit-flips from radiation or voltage drops) and hard errors (permanent defects like Stuck-at-Faults in RRAM). Traditionally, engineers have relied on hardware redundancy, such as Triple Modular Redundancy (TMR), to mitigate these risks. However, TMR can double or triple power and area costs, which is often prohibitive for resource-constrained edge devices.
FTT-NAS (Fault-Tolerant Neural Architecture Search) proposes a different approach: designing the neural network's architecture itself to be naturally resilient to hardware faults.
Hardware Fault Modeling
To search for resilient architectures, the authors first formalized how hardware faults manifest at the application level. They focused on two primary models:
- MiBB (MAC-i.i.d Bit-Bias): This models feature-map faults in FPGAs and ASICs, where errors accumulate within the adder trees of Multiply-Accumulate (MAC) units.
- adSAF (Arbitrary-distributed Stuck-at-Fault): This models weight faults in emerging memory technologies like Resistive RAM (RRAM), where cells become permanently stuck at specific values.
As shown in the hardware landscape above, different components exhibit different error characteristics. While some parts like control logic are difficult to model algorithmically, the computational logic and buffers can be effectively simulated using statistical bit-flip and bit-bias models.
The FTT-NAS Framework
The core of FTT-NAS is an automated search process that optimizes for both "clean" accuracy and "faulty" accuracy. The search objective is defined by a multi-objective reward: $Reward = (1 - \alpha) \cdot acc_{clean} + \alpha \cdot acc_{faulty}$.
A critical component of this framework is Fault-Tolerant Training (FTT). During the search, random faults are injected into the "super-net"—the large search space from which specific architectures are sampled. This ensures that the weights learned during the search process are inherently adapted to noisy hardware environments.
The Fragility of Standard Architectures
The motivation for FTT-NAS is underscored by the extreme fragility of popular manually designed architectures. When subjected to even low rates of hardware faults, the performance of models like MobileNet-V2 degrades rapidly.
As seen in the table, while ResNet-18 maintains some level of functionality at a $10^{-5}$ fault rate, MobileNet-V2 collapses to near-random guessing (10%) at a $10^{-4}$ fault rate. This highlights that efficiency-focused designs often sacrifice robustness.
Key Insights and Results
The search process revealed several architectural "rules of thumb" for fault tolerance:
- The "Fewer MACs" Rule: For feature-map faults, the controller preferred operations like Separable and Dilated Convolutions. By reducing the number of MAC operations per output pixel, these layers provide fewer opportunities for bit-flips to occur in the hardware adder tree.
- The "Small Weights" Rule: For RRAM weight faults, the search favored ReLU-Conv-BN blocks. These structures tend to produce smaller weight magnitudes, which minimizes the impact of a "stuck-at-maximum" fault.
The resulting models, F-FTT-Net and W-FTT-Net, demonstrated significant gains. Under a high feature fault rate ($3 \times 10^{-4}$), F-FTT-Net achieved 86.2% accuracy on CIFAR-10, whereas MobileNet-V2 failed completely. Similarly, under an 8% weight fault rate in RRAM, W-FTT-Net reached 69.6% accuracy, significantly outperforming ResNet-18 (60.8%).
Conclusion and Future Work
FTT-NAS demonstrates that architectural design is a powerful lever for hardware reliability. By moving beyond simple redundancy and into the realm of algorithmic resilience, we can deploy robust AI in challenging edge environments without the typical overhead of TMR. Future research will look toward scaling these findings to larger datasets like ImageNet and addressing the more complex challenge of faults in hardware control logic.
Summary updated