Abstract
Analogue memory-based deep neural networks provide energy-efficiency and per-area throughput gains relative to state-of-the-art digital counterparts such as graphics processing units. Recent advances focus largely on hardware-aware algorithmic training and improvements to circuits, architectures, and memory devices. Optimal translation of software-trained weights into analogue hardware weights—given the plethora of complex memory non-idealities—represents an equally important task. We report a generalised computational framework that automates the crafting of complex weight programming strategies to minimise accuracy degradations during inference, particularly over time. The framework is agnostic to network structure and generalises well across recurrent, convolutional, and transformer neural networks. As a highly flexible numerical heuristic, the approach accommodates arbitrary device-level complexity, making it potentially relevant for a variety of analogue memories. By quantifying the limit of achievable inference accuracy, it also enables analogue memory-based deep neural network accelerators to reach their full inference potential.
Introduction
The demand for energy-efficient AI hardware has led to a resurgence of interest in analogue memory-based accelerators. By performing computations directly within memory—using technologies like Phase-Change Memory (PCM)—these systems bypass the "von Neumann bottleneck" that limits traditional digital hardware like GPUs. However, the transition from software-trained weights to physical hardware conductances is fraught with challenges due to the inherent stochasticity and instability of analogue devices.
The Challenge of Analogue Hardware
Analogue memory devices are subject to several non-idealities that can degrade the accuracy of a neural network:
- Programming Errors: The inability to set a device to an exact target conductance.
- Conductance Drift: A temporal phenomenon where conductance values typically decrease over time.
- Read Noise: Fluctuations in the measured value during each inference operation.
Naive programming strategies, which often map weights to conductances in a straightforward manner, fail to account for how these errors compound over time. As devices drift at different rates, the fidelity of the programmed weights degrades, leading to a rapid decline in inference performance.
The Differential Weight Evolution (DWE) Framework
To address these challenges, researchers have developed the Differential Weight Evolution (DWE) framework. This generalized computational approach automates the creation of optimized weight programming strategies.
The framework incorporates several key innovations:
- Proxy Error Metric: To avoid the computational burden of full-scale inference simulations, DWE utilizes a time-averaged, normalized mean-squared error (MSE) metric. This serves as a fast and effective proxy for long-term accuracy preservation.
- Dimensionality Reduction: Rather than optimizing millions of individual weights, the framework optimizes a set of "discretized" weight values across the expected distribution. Linear interpolation is then applied to map all weights in the network.
- Hardware-Aware (HWA) Training Integration: DWE acts as a critical final step following HWA training. While HWA training makes the network resilient to noise, DWE determines the mathematically optimal way to load those weights onto the physical hardware.
Experimental Validation
The DWE framework was validated across three diverse benchmarks to demonstrate its versatility:
- Language Modeling: A two-layer Long Short-Term Memory (LSTM) network using the Penn Treebank dataset.
- Image Classification: A ResNet-32 convolutional neural network (CNN) tested on CIFAR-10.
- Natural Language Inference: The BERT-base transformer model evaluated on the MNLI dataset.
Results and Discussion
The optimized strategies generated by DWE significantly outperformed naive methods across all benchmarks.
Key findings include:
- Substantial Error Reduction: Optimized strategies reduced weight errors by approximately 39% immediately after programming and 17% after one month of simulated operation.
- Enhanced Long-Term Stability: For the BERT-base model, which is particularly sensitive to noise, the DWE-optimized strategy maintained performance near the theoretical peak for over $10^7$ seconds (approximately 4 months), whereas naive programming led to a total collapse in accuracy.
- Emergent Redundancy: The optimizer often discovered counter-intuitive strategies, such as programming small non-zero values into "negative" conductance devices to represent positive weights. This redundancy helps mitigate the impact of stochastic drift.
Conclusion
The DWE framework represents a vital "missing link" in the development of analogue AI accelerators. By providing a hardware-agnostic method to optimize weight programming, it enables these systems to achieve near-digital accuracy over extended periods. Future work may explore local optimization for individual crossbar arrays and the inclusion of energy consumption as an optimization constraint.
Summary updated