VLSI Implementation of a High-Capacity Neural Network Associative Memory

Tzi-Dar Chiueh, Rodney M. GoodmanNeurIPS 1989

Introduction

Associative memories are neural networks designed to "remember" patterns. When provided with a noisy or incomplete input, they recall the closest stored memory. The classic Hopfield network is a famous example, but it suffers from a major flaw: its storage capacity scales only linearly with the length of the patterns.

To increase capacity, researchers proposed "high-order" correlation memories. While mathematically powerful, these are notoriously difficult to implement in hardware because the number of required connections usually explodes. Researchers at Caltech tackled this by developing a custom VLSI chip that implements an "Exponential Correlation Associative Memory" (ECAM). By leveraging the natural sub-threshold physics of MOS transistors to perform exponential calculations, this chip achieves significantly higher storage capacity while avoiding the connection explosion problem.

Hardware Implementation of the ECAM Model

The authors designed a prototype chip using a 3$\mu$m CMOS process. The core innovation lies in mapping the ECAM evolution equation directly onto silicon physics. The chip is fully programmable, containing a 32 x 24 array of ECAM cells integrated with standard RAM cells for pattern storage.

Core ECAM cell circuit design

The implementation relies on three main components:

  1. Correlation Computation: A voltage-divider circuit using NMOS transistors and XOR gates compares the input to stored patterns. The output voltage is proportional to how well the patterns match.
  2. The "Exponential" Trick: Instead of using complex digital logic to calculate exponents, the authors use NMOS transistors operating in the sub-threshold region. In this state, the transistor's drain current is naturally an exponential function of its gate voltage. This allows the chip to perform a sophisticated non-linear calculation almost "for free."
  3. Summation and Thresholding: The chip uses current summing to aggregate the results and a simple differential amplifier to act as the "threshold neuron," deciding whether the final output bit should be a +1 or -1.

Evaluation and Key Insights

The authors evaluated the chip using random 24-bit patterns to test error-correction limits and applied it to $4 \times 4$ blocks of binary images to demonstrate real-world potential for image compression and recognition.

Physics as Computation

The most elegant idea in this research is using the sub-threshold characteristics of a transistor to implement exponentiation. This is a hallmark of "neuromorphic" engineering—using the inherent physical properties of the medium to perform complex mathematical operations efficiently.

The $V_{BB}$ Sweet Spot

During testing, the authors discovered an interesting trade-off regarding the bias voltage ($V_{BB}$).

Error correction performance across different bias voltages

The chip performed best when $V_{BB}$ was set to 2V, which is actually above the threshold voltage. At 2V, the transistors move out of the pure exponential (sub-threshold) region and into a "hybrid" square-law region. While 1V provides a "purer" mathematical exponential function, 2V provides a much larger dynamic range. The increased signal strength at 2V outweighed the mathematical purity of the 1V setting, leading to significantly better error correction in practice.

Results and Future Work

  • Speed and Throughput: The prototype chip performs a single associative recall in under 3 microseconds. For image processing, this translates to processing a $512 \times 512$ binary image in about 49 ms, or roughly 20 images per second.
  • Robustness: The hardware implementation proved to be nearly as effective as an ideal computer simulation. The iterative approach used here is more robust than "one-shot" winner-take-all circuits commonly used in other associative memories.

While the prototype is small (handling only 24-bit patterns), the architecture is well-suited for high-speed tasks like optical character recognition (OCR) and real-time vector quantization for video. Future production-grade chips will need to address the limits of precision and power as the number of patterns and bits scale up, as well as minor non-linearities in the NMOS transistors used in the correlation circuits.

Summary updated

BibTeX
@inproceedings{chiueh1989vlsi,
  author = {Tzi-Dar Chiueh and Rodney M. Goodman},
  title = {{VLSI Implementation of a High-Capacity Neural Network Associative Memory}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1989},
}
Figure