Abstract
This paper proposes a novel hardware synapse using Spin-Transfer Torque Magnetic Tunnel Junctions (STT-MTJs) to facilitate efficient, in-situ training of Quantized Neural Networks (QNNs). By exploiting the intrinsic stochastic switching behavior of MTJs, the proposed architecture eliminates the need for external random number generators and high-precision weight storage. The design utilizes a 4T2R ternary synapse structure that supports analog-domain feedforward and backpropagation operations. System-level simulations on MNIST, SVHN, and CIFAR-10 datasets demonstrate that the MTJ-based hardware achieves near-software accuracy with significant energy efficiency improvements, reaching up to 18.3 TOPs/W for feedforward operations.
Introduction
Deep Neural Networks (DNNs) are increasingly difficult to deploy on resource-constrained edge devices due to their substantial power and memory demands. Quantized Neural Networks (QNNs), which utilize low-bitwidth weights such as binary (-1, 1) or ternary (-1, 0, 1), offer a promising path toward reducing memory footprints. However, training QNNs remains computationally expensive, often requiring high-precision (32-bit) weights for gradient accumulation and complex stochastic update rules to maintain accuracy. Current hardware implementations typically rely on external Pseudo-Random Number Generators (PRNGs), which introduce significant area and power overhead.
Methodology
The researchers introduce a hardware implementation of the GXNOR framework using STT-MTJs, focusing on three key architectural innovations:
1. Stochasticity as a Feature
Rather than treating the probabilistic switching of MTJs as a reliability drawback, this design leverages it as a functional feature. In the high-current regime, the probability of an MTJ switching its state is a direct function of the voltage pulse duration. This allows the MTJ to serve as its own "stochastic updater," removing the need for digital PRNG circuits.
2. The 4T2R Synapse Architecture
The proposed ternary synapse consists of four transistors and two MTJs (4T2R). By configuring the resistance states of the two MTJs, the circuit can represent four distinct states, which are mapped to the ternary values -1, 0, and 1.
As shown in the mapping table below, the output current $I_{out}$ is determined by the differential resistance of $R_1$ and $R_2$, enabling the representation of zero states through balanced resistances.
3. In-Situ Training and Processing Near Memory
The architecture supports Processing Near Memory (PNM), where feedforward and backpropagation operations are performed in the analog domain using Kirchhoff’s Current Law (KCL) within the synapse array. Weight updates occur directly within the array via specific voltage pulses that trigger stochastic switching based on calculated errors, significantly reducing data movement energy.
Experimental Evaluation
The architecture was characterized using SPICE simulations (GlobalFoundries 28nm FD-SOI process) and evaluated on standard computer vision benchmarks: MNIST, SVHN, and CIFAR-10. The performance was compared against ideal software implementations of Binary Neural Networks (BNNs), Ternary Neural Networks (TNNs), and the original GXNOR algorithm.
Key Results
The MTJ-based hardware demonstrated performance remarkably close to software-based training:
- Accuracy: Achieved 98.61% on MNIST, 93.99% on SVHN, and 83.02% on CIFAR-10, trailing software benchmarks by less than 1%.
- Energy Efficiency: The synapse array showed a potential efficiency of 18.3 TOPs/W for feedforward operations and 3 TOPs/W for weight updates.
- Robustness: The system proved resilient to device-to-device variations; even with a 30% relative standard deviation in resistance, MNIST accuracy dropped by less than 0.5%.
Discussion and Conclusion
While the MTJ-based synapse array is highly efficient, the overall system performance is currently limited by peripheral circuits, particularly Analog-to-Digital Converters (ADCs), which consume a significant portion of total power. Additionally, cumulative leakage current from "zero" state synapses in very large arrays remains a challenge for KCL-based summation.
In conclusion, leveraging the physical properties of STT-MTJs for stochastic updates provides a pathway to highly efficient, in-situ trainable QNN hardware. Future work will focus on full SoC integration and optimizing peripheral control logic to handle complex stochastic pulse timing.
Summary updated