Long short-term memory networks in memristor crossbars

Can Li, Zhongrui Wang, Mingyi Rao, Daniel Belkin, Wenhao Song, Hao Jiang, Peng Yan, Yunning Li, Peng Lin, Miao Hu, Ning Ge, John Paul Strachan, Mark Barnell, Qing Wu, R. Stanley Williams, J. Joshua Yang, Qiangfei Xia

Abstract

Recent breakthroughs in recurrent deep neural networks with long short-term memory (LSTM) units has led to major advances in artificial intelligence. State-of-the-art LSTM models with significantly increased complexity and a large number of parameters, however, have a bottleneck in computing power resulting from limited memory capacity and data communication bandwidth. Here we demonstrate experimentally that LSTM can be implemented with a memristor crossbar, which has a small circuit footprint to store a large number of parameters and in-memory computing capability that circumvents the 'von Neumann bottleneck'. We illustrate the capability of our system by solving real-world problems in regression and classification, which shows that memristor LSTM is a promising low-power and low-latency hardware platform for edge inference.

Introduction

Long Short-Term Memory (LSTM) networks have become a cornerstone of modern sequence modeling, enabling breakthroughs in speech recognition, language translation, and time-series forecasting. However, as these models grow in complexity, they encounter a fundamental physical limit known as the "von Neumann bottleneck." In traditional computing architectures, the constant movement of massive weight matrices between separate memory and processing units consumes the majority of the system's energy and time. This "memory wall" is particularly restrictive for edge computing applications, where low latency and power efficiency are critical.

To address this, researchers have turned to in-memory computing. By utilizing memristor crossbars, computation can be performed directly where the data is stored, effectively eliminating the need for data transport. This paper presents the first experimental demonstration of an LSTM network implemented using a memristor-based hardware-software hybrid system.

Hardware Implementation and Methodology

The core of the experimental system is a 128×64 1T1R (one-transistor one-memristor) crossbar array. The memristors are composed of a Tantalum/Hafnium Oxide (Ta/HfO2) stack integrated onto a commercial transistor array.

In this architecture, neural network weights are mapped to the electrical conductance of the memristors. The system leverages fundamental physical laws to perform Matrix-Vector Multiplication (MVM)—the most computationally intensive part of the LSTM algorithm—in a single step:

  1. Ohm’s Law ($I = V \times G$): Performs the multiplication of input voltages by device conductances.
  2. Kirchhoff’s Current Law: Sums the resulting currents along the columns of the crossbar.

To represent both positive and negative weights, the researchers employed a differential encoding scheme where each weight is represented by the difference in conductance between two memristors.

Overcoming Hardware Non-Idealities

A significant challenge in memristive systems is device-to-device variation and non-linearities. To mitigate these issues, the team utilized in-situ training via the Back-Propagation Through Time (BPTT) algorithm. By training the network directly on the hardware, the model learns to compensate for the specific physical characteristics and defects of the crossbar. This approach ensures that the network remains robust despite the inherent imperfections of analog hardware.

Experimental Results

The memristor-based LSTM was evaluated on two distinct tasks:

1. Time-Series Regression: Airline Passenger Prediction

The system was tasked with predicting future passenger numbers based on the International Airline Passengers dataset (1949–1960). After 800 training epochs, the network successfully captured both the seasonal fluctuations and the long-term growth trends of the data, demonstrating its ability to handle regression tasks with high precision.

2. Temporal Classification: Gait Recognition

Using the USF-NIST Gait Dataset, the system attempted to identify eight individuals based on video sequences of their walking patterns. This task requires the network to maintain a "memory" of previous frames to distinguish between different gaits. The hardware implementation achieved a maximum classification accuracy of 79.1%, a performance level that closely matches idealized, defect-free software simulations.

Discussion and Future Directions

While this work represents a major milestone, the current implementation is a hybrid system. While MVM operations occur in the memristor hardware, the non-linear activation functions (sigmoid and tanh) and the gating logic are currently processed in software.

The next step toward a fully optimized edge inference engine involves integrating these peripheral circuits—such as analog-to-digital converters and non-linear function generators—directly onto the chip. Such integration would further reduce latency and power consumption by minimizing the need for data conversion.

Conclusion

The experimental implementation of LSTMs in memristor crossbars proves that in-memory computing is a viable path forward for sequence modeling. By circumventing the von Neumann bottleneck and demonstrating resilience to hardware variations through in-situ training, this architecture provides a blueprint for the next generation of low-power, high-speed AI hardware at the edge.

Summary updated

Abstract

Recent breakthroughs in recurrent deep neural networks with long short-term memory (LSTM) units has led to major advances in artificial intelligence. State-of-the-art LSTM models with significantly increased complexity and a large number of parameters, however, have a bottleneck in computing power resulting from limited memory capacity and data communication bandwidth. Here we demonstrate experimentally that LSTM can be implemented with a memristor crossbar, which has a small circuit footprint to store a large number of parameters and in-memory computing capability that circumvents the 'von Neumann bottleneck'. We illustrate the capability of our system by solving real-world problems in regression and classification, which shows that memristor LSTM is a promising low-power and low-latency hardware platform for edge inference.
BibTeX
@inproceedings{li2019long,
  author = {Can Li and Zhongrui Wang and Mingyi Rao and Daniel Belkin and Wenhao Song and Hao Jiang and Peng Yan and Yunning Li and Peng Lin and Miao Hu and Ning Ge and John Paul Strachan and Mark Barnell and Qing Wu and R. Stanley Williams and J. Joshua Yang and Qiangfei Xia},
  title = {{Long short-term memory networks in memristor crossbars}},
  booktitle = {Nat. Mach. Intell},
  year = {2019},
  url = {https://openreview.net/forum?id=ldyLggyxKv},
}
Figure