Variance-aware weight quantization of multi-level resistive switching devices based on Pt/LaAlO3/SrTiO3 heterostructures

Sunwoo Lee, Jaeyoung Jeon, Kitae Eom, Chaehwa Jeong, Yongsoo Yang, Ji‐Yong Park, Chang‐Beom Eom, Hyung-Woo LeeScientific 2022 · DOI

TL;DR

This paper presents a novel Pt/LaAlO₃/SrTiO₃ (Pt/LAO/STO) heterostructure memristor capable of reliably storing over 27 distinct conductance states. To overcome the inherent state-overlap issue in such devices and improve their accuracy in neuromorphic applications, the authors introduce a "variance-aware weight quantization" (VAQ) method, which significantly enhances computational performance.

What Problem Does This Paper Address?

Developing efficient and accurate neuromorphic computing systems relies heavily on memristors that can emulate biological synapses. A key requirement for synaptic emulation is the ability to store multiple non-volatile conductance states with high precision, mimicking the graded nature of synaptic weights. However, existing resistive switching devices often suffer from variations in their conductance, leading to a "state-overlap issue." This overlap means that different intended weight values can be mapped to the same or very similar conductance states, leading to inaccurate weight quantization and reduced computational accuracy. The paper addresses this by first creating a more robust multi-level memristor and then proposing a novel quantization strategy to leverage its capabilities effectively.

Methods

The core of the paper involves two main components:

  1. Novel Memristor Device Fabrication: The researchers developed a Pt/LAO/STO heterostructure memristor. The key innovation lies in precisely controlling the tunneling of a two-dimensional electron gas (2DEG) across an ultrathin LAO barrier by redistributing surface oxygen vacancies. This controlled redistribution allows for the reliable programming of multiple, tunable conductance states (over 27). The fabrication process involved pulsed laser deposition (PLD) for growing the LAO film on a STO substrate, followed by Pt electrode deposition.
  2. Variance-Aware Weight Quantization (VAQ): To address the state-overlap issue, the authors propose the VAQ method. Unlike conventional uniform quantization, where conductance states are evenly spaced, VAQ defines non-uniform conductance states. These states are strategically spaced to minimize overlap between their distributions. This means that states with similar conductance values are placed closer together, while states with larger differences are spaced further apart, effectively creating "finer-grained" resolution in regions where data is more concentrated and "coarser-grained" resolution where data is sparser. This approach is implemented by pre-defining optimal writing voltages for the memristor.

Datasets

The paper uses simulated data derived from neural network training for evaluation:

  • ResNet20 output layer data: Data was collected from a ResNet20 neural network trained on the CIFAR-10 dataset. This includes histograms of input data (activations) and weight values, which are noted to have a normal-like distribution.
  • Fashion-MNIST dataset: Used for evaluating the convolution operation performance. A sample image (#1888) from this dataset was used to demonstrate the VAQ's effectiveness in convolution operations.

Key Insights and Ideas

  • Oxygen Vacancy Engineering for Precision: The precise control over oxygen vacancy distribution is crucial for achieving reliable multi-level switching in oxide heterostructures. This offers a pathway to engineer memristor behavior beyond simple on/off states.
  • Non-Uniform Quantization is Key: The paper strongly suggests that uniform quantization is suboptimal for memristor-based computing, especially when dealing with data that exhibits non-uniform distributions (like normal-like distributions of weights and activations).
  • VAQ Mitigates State Overlap: The VAQ method directly tackles the state-overlap problem by intelligently designing the quantization levels. This is not just a theoretical improvement; simulations show a significant reduction in quantization errors.
  • VAQ Aligns with Data Distribution: The VAQ's non-uniform state distribution naturally aligns with the typical normal-like distribution of weights and activations in neural networks, leading to better representation of these values.
  • Practical Implementation: The VAQ method can be implemented by simply pre-defining the writing voltages for the memristor, making it a practical strategy for existing resistive switching devices without requiring complex external circuitry.

Key Results

  • Multi-Level Conductance: The Pt/LAO/STO memristor reliably achieved over 27 distinct conductance states with a high on/off ratio (2.84×10³%).
  • Reduced Quantization Error: Simulations demonstrated that VAQ significantly reduces quantization errors compared to uniform quantization. For example, in simulations using 12 states, VAQ showed a substantial decrease in measurement error, particularly in the higher conductance regimes where uniform quantization struggled.
  • Improved Computational Accuracy: When applied to matrix-matrix multiplication tasks, VAQ led to more accurate output activations compared to uniform quantization. This was further validated in a convolution operation simulation using the Fashion-MNIST dataset, where VAQ produced an output image much closer to the original, unquantized output.
  • Linearity: The conductance change with respect to the writing voltage was found to be desirably linear, which is beneficial for efficient and accurate neural network training.

Limitations and Future Work

  • Limited VAQ States for Evaluation: For proof-of-concept demonstrations, the authors selected a subset of 12 VAQ states out of the total 27 available states. While this subset effectively highlighted the VAQ's advantages, it doesn't represent the full potential performance of the 2DEG memristors with all their states.
  • Activation Functions: The simulations did not explicitly incorporate specific activation functions (like sigmoid, ReLU). The authors note that the VAQ's advantage would likely be even more pronounced with such functions, as they can further shift data distributions towards smaller values, exacerbating the information loss in uniform quantization.
  • Future Work: The paper suggests that the 2DEG memristors can implement an even larger number of VAQ states, leading to higher performance. The VAQ method itself is presented as a stepping stone for developing high-performance multi-bit memristors based on complex oxide heterostructures and other novel materials. The authors believe the VAQ is readily applicable to other conventional resistive switching devices.

Summary updated

Abstract

Resistive switching devices have been regarded as a promising candidate of multi-bit memristors for synaptic applications. The key functionality of the memristors is to realize multiple non-volatile conductance states with high precision. However, the variation of device conductance inevitably causes the state-overlap issue, limiting the number of available states. The insufficient number of states and the resultant inaccurate weight quantization are bottlenecks in developing practical memristors. Herein, we demonstrate a resistive switching device based on Pt/LaAlO<sub>3</sub>/SrTiO<sub>3</sub> (Pt/LAO/STO) heterostructures, which is suitable for multi-level memristive applications. By redistributing the surface oxygen vacancies, we precisely control the tunneling of two-dimensional electron gas (2DEG) through the ultrathin LAO barrier, achieving multiple and tunable conductance states (over 27) in a non-volatile way. To further improve the multi-level switching performance, we propose a variance-aware weight quantization (VAQ) method. Our simulation studies verify that the VAQ effectively reduces the state-overlap issue of the resistive switching device. We also find that the VAQ states can better represent the normal-like data distribution and, thus, significantly improve the computing accuracy of the device. Our results provide valuable insight into developing high-precision multi-bit memristors based on complex oxide heterostructures for neuromorphic applications.
BibTeX
@inproceedings{lee2022variance,
  author = {Sunwoo Lee and Jaeyoung Jeon and Kitae Eom and Chaehwa Jeong and Yongsoo Yang and Ji‐Yong Park and Chang‐Beom Eom and Hyung-Woo Lee},
  title = {{Variance-aware weight quantization of multi-level resistive switching devices based on Pt/LaAlO3/SrTiO3 heterostructures}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-13121-4},
}
Figure