MTJ-Based Hardware Synapse Design for Quantized Deep Neural Networks

Tzofnat Greenberg-Toledo, Ben Perach, Daniel Soudry, Shahar Kvatinsky

Abstract

This paper proposes a novel hardware synapse using Spin-Transfer Torque Magnetic Tunnel Junctions (STT-MTJs) to facilitate efficient, in-situ training of Quantized Neural Networks (QNNs). By exploiting the intrinsic stochastic switching behavior of MTJs, the proposed architecture eliminates the need for external random number generators and high-precision weight storage. The design utilizes a 4T2R ternary synapse structure that supports analog-domain feedforward and backpropagation operations. System-level simulations on MNIST, SVHN, and CIFAR-10 datasets demonstrate that the MTJ-based hardware achieves near-software accuracy with significant energy efficiency improvements, reaching up to 18.3 TOPs/W for feedforward operations.

Introduction

Deep Neural Networks (DNNs) are increasingly difficult to deploy on resource-constrained edge devices due to their substantial power and memory demands. Quantized Neural Networks (QNNs), which utilize low-bitwidth weights such as binary (-1, 1) or ternary (-1, 0, 1), offer a promising path toward reducing memory footprints. However, training QNNs remains computationally expensive, often requiring high-precision (32-bit) weights for gradient accumulation and complex stochastic update rules to maintain accuracy. Current hardware implementations typically rely on external Pseudo-Random Number Generators (PRNGs), which introduce significant area and power overhead.

Methodology

The researchers introduce a hardware implementation of the GXNOR framework using STT-MTJs, focusing on three key architectural innovations:

1. Stochasticity as a Feature

Rather than treating the probabilistic switching of MTJs as a reliability drawback, this design leverages it as a functional feature. In the high-current regime, the probability of an MTJ switching its state is a direct function of the voltage pulse duration. This allows the MTJ to serve as its own "stochastic updater," removing the need for digital PRNG circuits.

2. The 4T2R Synapse Architecture

The proposed ternary synapse consists of four transistors and two MTJs (4T2R). By configuring the resistance states of the two MTJs, the circuit can represent four distinct states, which are mapped to the ternary values -1, 0, and 1.

MTJ-based synapse structure showing the free and fixed layers and the dual-MTJ configuration.

As shown in the mapping table below, the output current $I_{out}$ is determined by the differential resistance of $R_1$ and $R_2$, enabling the representation of zero states through balanced resistances.

Mapping of ternary weights to MTJ resistance states ($R_{on}$, $R_{off}$) and the resulting output current.

3. In-Situ Training and Processing Near Memory

The architecture supports Processing Near Memory (PNM), where feedforward and backpropagation operations are performed in the analog domain using Kirchhoff’s Current Law (KCL) within the synapse array. Weight updates occur directly within the array via specific voltage pulses that trigger stochastic switching based on calculated errors, significantly reducing data movement energy.

Experimental Evaluation

The architecture was characterized using SPICE simulations (GlobalFoundries 28nm FD-SOI process) and evaluated on standard computer vision benchmarks: MNIST, SVHN, and CIFAR-10. The performance was compared against ideal software implementations of Binary Neural Networks (BNNs), Ternary Neural Networks (TNNs), and the original GXNOR algorithm.

Key Results

The MTJ-based hardware demonstrated performance remarkably close to software-based training:

  • Accuracy: Achieved 98.61% on MNIST, 93.99% on SVHN, and 83.02% on CIFAR-10, trailing software benchmarks by less than 1%.
  • Energy Efficiency: The synapse array showed a potential efficiency of 18.3 TOPs/W for feedforward operations and 3 TOPs/W for weight updates.
  • Robustness: The system proved resilient to device-to-device variations; even with a 30% relative standard deviation in resistance, MNIST accuracy dropped by less than 0.5%.

Discussion and Conclusion

While the MTJ-based synapse array is highly efficient, the overall system performance is currently limited by peripheral circuits, particularly Analog-to-Digital Converters (ADCs), which consume a significant portion of total power. Additionally, cumulative leakage current from "zero" state synapses in very large arrays remains a challenge for KCL-based summation.

In conclusion, leveraging the physical properties of STT-MTJs for stochastic updates provides a pathway to highly efficient, in-situ trainable QNN hardware. Future work will focus on full SoC integration and optimizing peripheral control logic to handle complex stochastic pulse timing.

Summary updated

BibTeX
@inproceedings{greenbergtoledo2021mtj,
  author = {Tzofnat Greenberg-Toledo and Ben Perach and Daniel Soudry and Shahar Kvatinsky},
  title = {{MTJ-Based Hardware Synapse Design for Quantized Deep Neural Networks}},
  booktitle = {Semiconductor},
  year = {2021},
  doi = {10.1088/1361-6641/ac251b},
}
Figure