Bit Error Robustness for Energy-Efficient DNN Accelerators

David Stutz, Nandhini Chandramoorthy, Matthias Hein, Bernt Schiele

TL;DR

This paper introduces a strategy to make Deep Neural Networks (DNNs) resilient to the random bit errors that occur when hardware accelerators are run at ultra-low voltages to save power. By combining a robust quantization scheme (RQUANT), aggressive weight clipping, and random bit error training (RANDBET), the authors demonstrate that models can maintain high accuracy even with a 1% bit error rate, enabling up to 30% energy savings in SRAM-heavy accelerators.

What Problem Does This Paper Address?

Energy efficiency is the "holy grail" of edge computing. To minimize power consumption, hardware designers often scale down the supply voltage ($V$) of the SRAM used to store DNN weights. Because dynamic power consumption is proportional to the square of the voltage ($P \propto V^2$), even small reductions lead to significant savings.

However, there is a floor called $V_{min}$. Scaling below this threshold causes the memory to become unreliable, leading to exponential increases in random bit-level failures. Standard error-correction codes (ECC) are often too "heavy" or expensive to handle the high error rates (e.g., $p=1\%$) required for massive energy savings. This paper asks: Can we design the DNN itself to be "error-aware" so it can run on "approximate" hardware without crashing?

Methods

The authors propose a three-layered defense:

  1. Robust Quantization (RQUANT): The authors found that standard fixed-point quantization is surprisingly fragile. Specifically, using signed integers is risky because a single bit flip in the Most Significant Bit (MSB) can flip a large positive weight to a large negative one. RQUANT uses per-layer asymmetric quantization into unsigned integers with proper rounding. This ensures that bit flips result in less catastrophic value changes.
  2. Weight Clipping: During training, weights are constrained to a tight range (e.g., $[-0.1, 0.1]$). While this might seem like it just reduces the "scale" of errors, the authors show it acts as a powerful regularizer. It forces the network to achieve high confidence by using more weights (redundancy) rather than a few high-magnitude weights.
  3. Random Bit Error Training (RANDBET): Similar to how "Dropout" or data augmentation improves generalization, RANDBET injects random bit flips into the quantized weights during the forward pass of training. The model effectively learns to find "flat" regions in the weight space where a few bit flips don't significantly alter the output.

Datasets

The researchers evaluated their methods on standard computer vision benchmarks:

  • MNIST: A baseline for sanity checks.
  • CIFAR-10 and CIFAR-100: Used to test more complex architectures like SimpleNet and ResNets.
  • Profiled SRAM Bit Error Patterns: Instead of just simulating random noise, the authors used error maps from real 14nm SRAM chips to prove their "random" training generalizes to real-world hardware defects.

Key Insights and Ideas

  • The "Sign Bit" Trap: In symmetric quantization, the sign bit is a single point of failure. By shifting to unsigned integer representation (RQUANT), the authors significantly dampened the impact of random flips.
  • Redundancy via Clipping: This is the most counter-intuitive finding. Clipping weights doesn't just make the errors smaller; it changes the strategy of the network. To produce a strong "logit" (prediction score) with small weights, the network must rely on many different weights contributing to the same result. If one bit flips, the other weights "vote it down."
  • Generalization vs. Specialization: Previous work tried training models on specific "error maps" of a particular chip. This paper shows that training on completely random bit errors (RANDBET) actually generalizes better across different chips and different operating voltages.

Key Results

  • Energy Savings: On CIFAR-10, the authors achieved 20% energy savings with less than 1% accuracy loss using 8-bit quantization.
  • Low Precision: Even with 4-bit DNNs, they achieved 30% energy savings at the cost of only 2.5% accuracy.
  • Robustness Floor: Without these techniques, a standard DNN's accuracy often drops to chance levels (10% on CIFAR-10) once the bit error rate hits 0.5%–1%. With the proposed methods, accuracy remains near-baseline (above 90%).
  • Hardware Agnostic: The RANDBET models performed well on real-world bit error patterns from different SRAM chips without ever having seen those specific patterns during training.

Limitations and Future Work

  • Training Overhead: RANDBET requires an additional forward and backward pass per training iteration to calculate gradients for both "clean" and "perturbed" weights, roughly doubling or tripling training time.
  • The Accuracy-Robustness Trade-off: There is a slight "clean" accuracy penalty. Models trained to be robust are often 1-2% less accurate on perfect hardware than their non-robust counterparts.
  • Extreme Low-Bit Quantization: While 4-bit results were strong, the authors noted that 2-bit models remain a challenge, as the limited precision makes it harder for the network to develop the necessary redundancy. Future work could explore more sophisticated quantization schemes specifically for 2-bit and 3-bit regimes.

Summary updated

Abstract

Deep neural network (DNN) accelerators received considerable attention in past years due to saved energy compared to mainstream hardware. Low-voltage operation of DNN accelerators allows to further reduce energy consumption significantly, however, causes bit-level failures in the memory storing the quantized DNN weights. In this paper, we show that a combination of robust fixed-point quantization, weight clipping, and random bit error training (RandBET) improves robustness against random bit errors in (quantized) DNN weights significantly. This leads to high energy savings from both low-voltage operation as well as low-precision quantization. Our approach generalizes across operating voltages and accelerators, as demonstrated on bit errors from profiled SRAM arrays. We also discuss why weight clipping alone is already a quite effective way to achieve robustness against bit errors. Moreover, we specifically discuss the involved trade-offs regarding accuracy, robustness and precision: Without losing more than 1% in accuracy compared to a normally trained 8-bit DNN, we can reduce energy consumption on CIFAR-10 by 20%. Higher energy savings of, e.g., 30%, are possible at the cost of 2.5% accuracy, even for 4-bit DNNs.
BibTeX
@inproceedings{stutz2021bit,
  author = {David Stutz and Nandhini Chandramoorthy and Matthias Hein and Bernt Schiele},
  title = {{Bit Error Robustness for Energy-Efficient DNN Accelerators}},
  booktitle = {MLSys},
  year = {2021},
}

Loading related content...

Figure