MemSE: Fast MSE Prediction for Noisy Memristor-Based DNN Accelerators

Jonathan Kern, Sébastien Henwood, Gonçalo Mordido, Elsa Dupraz, Abdeldjalil Aïssa-El-Bey, Yvon Savaria, François Leduc-Primeau

Introduction

As deep learning moves toward edge computing—powering applications in mobile devices, IoT sensors, and autonomous systems—the demand for energy-efficient hardware has never been higher. Memristor crossbars have emerged as a leading candidate for this transition. By performing Matrix-Vector Multiplication (MVM) directly in memory using physical laws (Ohm’s and Kirchhoff’s), they offer significant efficiency gains over traditional digital architectures.

However, memristors are inherently noisy. They suffer from programming noise (imprecision in setting resistance) and quantization errors due to limited bit-precision (typically 2 to 7.5 bits). Traditionally, assessing how this noise impacts a Deep Neural Network (DNN) requires thousands of Monte-Carlo simulations, where random noise is repeatedly sampled to observe its effect on accuracy. This computational bottleneck makes it difficult for designers to quickly optimize hardware parameters for a specific power budget. This paper introduces MemSE, a theoretical framework that replaces slow simulations with fast, analytical predictions.

The MemSE Framework

MemSE (Memristor Mean Squared Error) provides a mathematical shortcut to track how noise propagates through a neural network. Instead of simulating thousands of noisy iterations, the framework derives the statistical properties of the data as it flows through the model.

1. Noise Modeling

The framework begins by decomposing the conductance ($G$) of a memristor into three distinct components:

  • Ideal Value: The intended weight of the neural network.
  • Quantization Error: A deterministic error resulting from the limited precision of the hardware.
  • Programming Noise: A random component representing the physical variability of the device.

2. Moment Tracking and Taylor Expansions

To avoid the need for sampling, MemSE tracks the first and second statistical moments—the mean and variance—of the activations at each layer. While linear operations (like convolutions) are straightforward to handle mathematically, non-linear activation functions (such as Softplus) present a challenge. MemSE utilizes second-order Taylor expansions to approximate how the noise distribution is transformed after passing through these non-linearities, allowing the framework to maintain an accurate estimate of the error distribution throughout the entire network.

3. Hardware-Software Co-optimization

One of the most critical "knobs" in memristor design is the maximum conductance ($G_{max}$). Increasing $G_{max}$ reduces the relative impact of noise, thereby improving accuracy, but it also increases power consumption. MemSE formulates this as a constrained optimization problem. By using the fast MSE prediction as a proxy for accuracy, the researchers can employ a genetic algorithm to find the optimal scaling factors for each layer, maximizing performance while staying within a strict power limit.

Experimental Results

The researchers evaluated MemSE using the CIFAR-10 dataset and custom Convolutional Neural Networks (CNNs) of varying complexities. The results demonstrate three primary advantages:

  • Massive Computational Speedup: MemSE is approximately 85x faster than traditional Monte-Carlo simulations. For a standard CNN, the MSE computation took only 27ms, compared to 2.3 seconds for simulation-based methods. This speedup is transformative for design-space exploration.
  • High Predictive Accuracy: The analytical curves for MSE closely match the results of empirical simulations, particularly in the high-accuracy regimes required for real-world deployment. The study confirmed that Mean Squared Error at the final layer serves as a reliable predictor of final classification accuracy.
  • Effective Optimization: The framework successfully identified hardware configurations that allow for different scaling factors across layers. This "layer-wise" flexibility allows the system to allocate more power to noise-sensitive layers while saving energy elsewhere, improving the overall power-accuracy tradeoff.

Discussion and Limitations

While MemSE represents a significant step forward, the authors identify specific boundaries for the framework. The accuracy of the Taylor expansion approximation begins to degrade in extremely high-noise scenarios where the variance is too large for local approximations to hold.

Furthermore, the current iteration focuses on specific architectural components like convolutions, linear layers, and Softplus activations. Future work is required to explicitly incorporate Batch Normalization and Residual connections (ResNets), as well as to explore the impact of non-differentiable activation functions like ReLU.

Conclusion

MemSE provides a robust theoretical foundation for the next generation of AI hardware. By enabling researchers to predict accuracy degradation in milliseconds rather than seconds, it paves the way for rapid hardware-software co-design, ensuring that edge AI devices can remain both precise and power-efficient.

Summary updated

Abstract

Memristors enable the computation of matrix-vector multiplications (MVM) in memory and, therefore, show great potential in highly increasing the energy efficiency of deep neural network (DNN) inference accelerators. However, computations in memristors suffer from hardware non-idealities and are subject to different sources of noise that may negatively impact system performance. In this work, we theoretically analyze the mean squared error of DNNs that use memristor crossbars to compute MVM. We take into account both the quantization noise, due to the necessity of reducing the DNN model size, and the programming noise, stemming from the variability during the programming of the memristance value. Simulations on pre-trained DNN models showcase the accuracy of the analytical prediction. Furthermore the proposed method is almost two order of magnitude faster than Monte-Carlo simulation, thus making it possible to optimize the implementation parameters to achieve minimal error for a given power constraint.
BibTeX
@inproceedings{kern2022memse,
  author = {Jonathan Kern and Sébastien Henwood and Gonçalo Mordido and Elsa Dupraz and Abdeldjalil Aïssa-El-Bey and Yvon Savaria and François Leduc-Primeau},
  title = {{MemSE: Fast MSE Prediction for Noisy Memristor-Based DNN Accelerators}},
  booktitle = {AICAS},
  year = {2022},
  url = {https://openreview.net/forum?id=t1GR0yjVP4},
}
Figure