Abstract
Deep neural network (DNN) accelerators received considerable attention in past years due to saved energy compared to mainstream hardware. Low-voltage operation of DNN accelerators allows to further reduce energy consumption significantly, however, causes bit-level failures in the memory storing the quantized DNN weights. In this paper, we show that a combination of robust fixed-point quantization, weight clipping, and random bit error training (RandBET) improves robustness against random bit errors in (quantized) DNN weights significantly. This leads to high energy savings from both low-voltage operation as well as low-precision quantization. Our approach generalizes across operating voltages and accelerators, as demonstrated on bit errors from profiled SRAM arrays. We also discuss why weight clipping alone is already a quite effective way to achieve robustness against bit errors. Moreover, we specifically discuss the involved trade-offs regarding accuracy, robustness and precision: Without losing more than 1% in accuracy compared to a normally trained 8-bit DNN, we can reduce energy consumption on CIFAR-10 by 20%. Higher energy savings of, e.g., 30%, are possible at the cost of 2.5% accuracy, even for 4-bit DNNs.
Introduction
Energy efficiency is a critical requirement for deploying Deep Neural Networks (DNNs) on edge devices such as smartphones and IoT sensors. One of the most effective strategies for reducing energy consumption is lowering the supply voltage of hardware accelerators. However, this approach faces a physical bottleneck: as the voltage drops below a critical threshold ($V_{min}$), the Static Random-Access Memory (SRAM) used for weight storage becomes unreliable. This results in random bit flips, where bits in the quantized weights transition between 0 and 1.
Standard DNNs are highly sensitive to these errors. A single bit flip in a high-significance position can transform a small weight into a massive outlier, catastrophic for model performance. While hardware-based solutions like error-correcting codes (ECC) exist, they introduce significant overhead. This research proposes a software-based approach to make DNNs inherently resilient to bit errors across various hardware and voltage levels.
Methodology: Building Robustness
The authors introduce a three-pronged strategy to enhance bit-error robustness:
- Robust Quantization (RQUANT): The choice of quantization scheme is pivotal. The researchers found that per-layer asymmetric quantization into unsigned integers, combined with proper rounding rather than simple truncation, significantly improves inherent robustness.
- Weight Clipping: By strictly constraining weights to a narrow range (e.g., $[-0.1, 0.1]$) during training, the network is prevented from relying on a few highly influential weights. This forces the model to distribute information across many weights, creating a form of functional redundancy.
- Random Bit Error Training (RandBET): This technique acts as weight-level data augmentation. During training, the model is exposed to random bit flips in its quantized weights. The optimization objective is modified to minimize loss under both clean and perturbed conditions, teaching the network to maintain stable outputs despite internal noise.
Key Results and Insights
The proposed methods were evaluated on MNIST, CIFAR-10, and CIFAR-100, as well as against real-world bit-error patterns from 14nm SRAM hardware.
Performance and Energy Savings
The results demonstrate a significant improvement in the robustness-energy trade-off:
- Energy Efficiency: On CIFAR-10, the robust models achieved a 20% reduction in SRAM energy with less than 1% accuracy loss compared to a standard 8-bit baseline.
- Aggressive Scaling: By tolerating a 2.5% accuracy drop, energy savings could be pushed to 30%, even with low-precision 4-bit models.
- Resilience to High Error Rates: At a 1% bit error rate, the proposed robust models maintained performance near their baseline, while standard models typically fail completely.
The Role of Redundancy
A key insight is that weight clipping is a powerful tool for robustness. By limiting individual weight magnitudes, the network must use more weights to achieve high-confidence predictions. This redundancy ensures that if a bit flip corrupts one weight, the collective signal from other weights remains largely intact.
Generalization
Unlike previous methods that required training on specific "error maps" for a particular chip, RandBET generalizes well. Models trained with random bit errors showed strong performance on real, non-random hardware errors, proving that the learned tolerance is a general property rather than an overfit to specific noise patterns.
Discussion and Future Work
While effective, these techniques come with trade-offs. RandBET increases training time due to the additional forward and backward passes required for error injection. Furthermore, the researchers noted that standard Batch Normalization (BN) is fragile under bit errors; switching to Group Normalization (GN) was necessary for stability. Future research may focus on optimizing training efficiency and exploring more sophisticated robust quantization schemes for extremely low-precision (e.g., 2-bit) models.
Summary updated