Abstract
Deep neural network (DNN) accelerators received considerable attention in past years due to saved energy compared to mainstream hardware. Low-voltage operation of DNN accelerators allows to further reduce energy consumption significantly, however, causes bit-level failures in the memory storing the quantized DNN weights. In this paper, we show that a combination of robust fixed-point quantization, weight clipping, and random bit error training (RandBET) improves robustness against random bit errors in (quantized) DNN weights significantly. This leads to high energy savings from both low-voltage operation as well as low-precision quantization. Our approach generalizes across operating voltages and accelerators, as demonstrated on bit errors from profiled SRAM arrays. We also discuss why weight clipping alone is already a quite effective way to achieve robustness against bit errors. Moreover, we specifically discuss the involved trade-offs regarding accuracy, robustness and precision: Without losing more than 1% in accuracy compared to a normally trained 8-bit DNN, we can reduce energy consumption on CIFAR-10 by 20%. Higher energy savings of, e.g., 30%, are possible at the cost of 2.5% accuracy, even for 4-bit DNNs.
Introduction
Energy efficiency is the primary constraint for deploying Deep Neural Networks (DNNs) on edge devices. While specialized hardware accelerators offer significant improvements over general-purpose processors, further gains can be achieved by scaling down the supply voltage ($V$). However, this approach faces a fundamental reliability barrier: as voltage drops below a certain threshold ($V_{min}$), the SRAM cells used for weight storage begin to exhibit random bit-level failures.
The Energy-Reliability Trade-off
The motivation for low-voltage operation is clear: dynamic power consumption scales quadratically with voltage ($P \propto V^2$). Unfortunately, as shown in Figure 2, reducing the supply voltage leads to an exponential increase in the bit error rate ($p$).
Standard error-correction codes (ECC) are often too resource-intensive to handle the high error rates (e.g., $p=1\%$) required for substantial energy savings. This research proposes an alternative: making the DNN itself inherently robust to these hardware-level faults, allowing it to run on "approximate" hardware without catastrophic failure.
Proposed Robustness Strategies
To enable reliable operation at ultra-low voltages, we introduce a three-layered defense strategy:
- Robust Quantization (RQUANT): Standard signed fixed-point quantization is fragile; a single flip in the Most Significant Bit (MSB) can invert a weight's sign, causing extreme value shifts. RQUANT utilizes per-layer asymmetric quantization into unsigned integers, ensuring that bit flips result in less severe value deviations.
- Weight Clipping: By constraining weights to a tight range (e.g., $[-0.1, 0.1]$) during training, the network is forced to develop redundancy. Instead of relying on a few high-magnitude weights, the model learns to distribute information across many smaller weights, creating a "voting" effect that mitigates the impact of individual bit flips.
- Random Bit Error Training (RANDBET): Similar to Dropout, RANDBET injects random bit flips into weights during the forward pass of training. This encourages the optimizer to find "flat" regions in the weight space where the loss function is less sensitive to small perturbations.
Experimental Results
The proposed methods were evaluated on benchmarks including MNIST, CIFAR-10, and CIFAR-100, using both simulated noise and real-world error patterns profiled from 14nm SRAM chips.
As illustrated in Figure 1, standard models ("NORMAL") fail rapidly as the bit error rate increases, often dropping to chance levels once $p$ hits 0.5%. In contrast, the combination of RQUANT, Clipping, and RANDBET maintains high accuracy even at a 1% bit error rate.
Key Findings
- Energy Savings: On CIFAR-10, we achieved 20% energy savings with less than 1% accuracy loss using 8-bit quantization. More aggressive 4-bit models reached 30% savings with a 2.5% accuracy trade-off.
- Generalization: Models trained with RANDBET on purely random errors generalized effectively to the non-uniform, hardware-specific error patterns of real SRAM chips without requiring chip-specific fine-tuning.
- The Clipping Effect: Weight clipping acts as a powerful regularizer, fundamentally changing the network's strategy to favor redundant representations over high-magnitude individual weights.
Conclusion
By integrating error-awareness into the training and quantization process, we can bypass the traditional $V_{min}$ limits of hardware accelerators. This approach enables significant energy savings—up to 30%—without requiring expensive hardware error correction, paving the way for more efficient AI at the edge.
Summary updated