A memristive deep belief neural network based on silicon synapses

Wei Wang, Loai Danial, Yang Li, Eric Herbelin, Evgeny Pikhay, Yakov Roizin, Barak Hoffer, Zhongrui Wang, Shahar Kvatinsky

TL;DR

Researchers have developed novel silicon-based memristive synapses that mimic biological neurons, enabling a more energy-efficient and compact deep belief neural network (DBN). This DBN achieves high accuracy on the MNIST dataset, demonstrating a promising hardware implementation for advanced AI tasks.

What Problem Does This Paper Address?

Traditional computing architectures face bottlenecks in moving data between memory and processing units, hindering the efficiency of deep neural networks (DNNs). Neuromorphic computing, which aims to mimic the brain's structure and function, offers a solution by performing computation within memory elements. However, existing memristive devices, while promising for synaptic emulation, often suffer from poor yield, limited reliable states, and non-ideal behaviors like non-linear conductance updates. This paper addresses these challenges by presenting a new type of silicon synapse fabricated using a standard CMOS process, which exhibits improved performance and reliability, and then demonstrates its application in a memristive deep belief network (DBN).

Methods

The core innovation lies in the development of two-terminal floating-gate memristive synaptic devices fabricated in a standard 180nm CMOS process. These devices leverage a floating gate to store charge, enabling analog tunability of conductance. Key features include:

  • CMOS Compatibility: Fabricated using a standard process without extra masks, facilitating integration.
  • Self-Selection: A non-linear I-V behavior at low voltages eliminates the need for separate selector devices in crossbar arrays, reducing sneak-path currents.
  • Analog Tunability: Conductance can be precisely adjusted through programming (depression) and erasing (potentiation) pulses.
  • High Endurance and Retention: Devices demonstrate reliable cycling and long-term data retention.
  • Mixed-Signal Design: The memristive arrays perform analog vector-matrix multiplications (VMMs) and stochastic sampling, while peripheral circuits handle contrastive divergence accumulation and communication in the digital domain. This approach simplifies peripheral circuitry by avoiding ADCs/DACs.
  • Modified Contrastive Divergence: A learning algorithm is employed that separates VMM and weight updates, making the training robust to non-linear weight updates and reducing write operations.

Datasets

The paper uses the modified National Institute of Standards and Technology (MNIST) dataset for training and testing the memristive deep belief neural network.

Key Insights and Ideas

  • CMOS-Friendly Synapses: The ability to fabricate reliable memristive synapses using standard CMOS processes is a significant step towards practical neuromorphic hardware.
  • Simplified Neural Circuits: By using binary neuron states and a mixed-signal approach, the peripheral circuitry for the RBM and DBN is greatly simplified, eliminating the need for complex ADCs/DACs.
  • Robust Training: The modified contrastive divergence algorithm effectively handles the non-linear weight update behavior of memristive devices, a common challenge in hardware implementations.
  • Energy Efficiency: The memristive array offers orders of magnitude higher energy efficiency for multiply-accumulate operations compared to GPUs.

Key Results

  • High Recognition Accuracy: The memristive DBN achieved a recognition accuracy of up to 97.05% on the MNIST dataset using 50-time sampling inference.
  • Energy Efficiency: The memristive array demonstrated an estimated 3.56 x 10^4 GOPS/W energy efficiency and 6.31 x 10^3 GOPS/mm² performance density for MAC operations, significantly outperforming GPUs.
  • Improved Yield and Reliability: The developed silicon synapses exhibit high yield, moderate device-to-device variations, and predictable cycling degradation.
  • DBN Performance: The memristive DBN demonstrated performance comparable to software solutions and state-of-the-art memristive DNNs that require more complex peripheral circuits.

Limitations and Future Work

  • Non-Ideal Behaviors: While mitigated, the paper acknowledges remaining non-ideal behaviors such as non-linear I-V reading and non-linear conductance updates.
  • Peripheral Circuitry: While simplified, the peripheral circuits are still necessary and their energy consumption and footprint are not included in the primary metrics.
  • Scaling: The authors suggest that further scaling down of the device dimensions would significantly improve performance metrics.
  • Future Work: The paper implies that further integration and optimization of these memristive DBNs could lead to even more powerful and efficient neuromorphic systems.

Summary updated

Abstract

Memristor-based neuromorphic computing could overcome the limitations of traditional von Neumann computing architectures -- in which data are shuffled between separate memory and processing units -- and improve the performance of deep neural networks. However, this will require accurate synaptic-like device performance, and memristors typically suffer from poor yield and a limited number of reliable conductance states. Here we report floating gate memristive synaptic devices that are fabricated in a commercial complementary metal-oxide-semiconductor (CMOS) process. These silicon synapses offer analogue tunability, high endurance, long retention times, predictable cycling degradation, moderate device-to-device variations, and high yield. They also provide two orders of magnitude higher energy efficiency for multiply-accumulate operations than graphics processing units. We use two 12-by-8 arrays of the memristive devices for in-situ training of a 19-by-8 memristive restricted Boltzmann machine for pattern recognition via a gradient descent algorithm based on contrastive divergence. We then create a memristive deep belief neural network consisting of three memristive restricted Boltzmann machines. We test this on the modified National Institute of Standards and Technology (MNIST) dataset, demonstrating recognition accuracy up to 97.05%.
BibTeX
@inproceedings{wang2022memristive,
  author = {Wei Wang and Loai Danial and Yang Li and Eric Herbelin and Evgeny Pikhay and Yakov Roizin and Barak Hoffer and Zhongrui Wang and Shahar Kvatinsky},
  title = {{A memristive deep belief neural network based on silicon synapses}},
  booktitle = {Nature Electronics},
  year = {2022},
  doi = {10.1038/s41928-022-00878-9},
}
Figure