Coupled neural associative memories.

Amin Karbasi, Amir Hesam Salavati, Amin Shokrollahi

TL;DR

This paper introduces a novel "coupled" neural associative memory architecture, inspired by the visual cortex, that significantly improves pattern recall accuracy in noisy conditions. By spatially coupling neural clusters across multiple planes, the model achieves exponentially larger storage capacities and superior noise resilience compared to prior methods.

What Problem Does This Paper Address?

Traditional artificial neural associative memories have struggled with a fundamental trade-off: either they can store a large number of patterns but perform poorly at recalling them in the presence of noise, or they can recall reliably but only store a limited number of patterns (often linearly scaling with the network size). This limitation hinders their practical application in scenarios where data is inherently noisy or incomplete. The authors aim to bridge this gap by developing an architecture that can both learn a vast number of patterns and robustly retrieve them even with significant noise.

Methods

The core innovation is a "coupled neural associative memory" architecture. This architecture divides neurons into local, overlapping clusters arranged in parallel "planes." Crucially, these planes are interconnected, creating a "coupled" structure that mimics aspects of the macaque brain's visual cortex. This spatial coupling is inspired by recent advancements in spatially-coupled codes, which have shown remarkable error-correction capabilities.

The learning phase involves training the network to memorize patterns by learning vectors orthogonal to sub-patterns within each cluster. The recall phase employs a two-tiered error correction algorithm:

  1. Intra-cluster correction: A local algorithm corrects errors within individual clusters using message-passing between pattern and constraint neurons.
  2. Inter-cluster/Inter-plane correction: A global algorithm coordinates error correction across clusters and planes, iteratively refining the recalled pattern.

The paper also provides a theoretical performance analysis using tools from coding theory, deriving bounds on the error probability and identifying thresholds for successful decoding.

Datasets

Not discussed in the paper. The authors mention generating patterns for simulations, but no specific pre-existing datasets are used or introduced.

Key Insights and Ideas

  • Spatial Coupling as a Key to Robustness: The paper strongly suggests that spatially coupling neural modules (clusters and planes) is a powerful mechanism for enhancing noise resilience and storage capacity in associative memories, mirroring successes in error-correcting codes.
  • Biomimicry Pays Off: The architectural inspiration from the visual cortex of the macaque brain appears to be a fruitful direction, leading to a model that is both biologically plausible and computationally effective.
  • Trade-off Reimagined: By moving beyond the assumption of memorizing arbitrary random patterns and focusing on patterns with inherent structure (redundancy in sub-patterns), the authors achieve a significant leap in both storage capacity and recall performance.
  • Bridging Theory and Practice: The paper effectively leverages theoretical tools from coding theory to analyze the performance of the neural architecture, providing analytical bounds that are then validated by simulations.

Key Results

  • The proposed coupled architecture demonstrates significantly better noise elimination capabilities compared to previous associative memory models.
  • The architecture supports learning an exponentially large number of patterns, a substantial improvement over prior linear scaling.
  • Simulations show that the coupled system achieves much better performance in eliminating noise during the recall phase.
  • Theoretical analysis provides thresholds for the maximum initial symbol error probability that the system can tolerate, showing a marked improvement with the coupled architecture (e.g., for e=2, the threshold increases from ~0.197 to ~0.394).

Limitations and Future Work

  • Learning Phase: While the paper focuses on the recall phase and assumes a learned connectivity graph, the authors mention that they are currently applying their proposed learning method to a database of natural images to assess its real-world performance.
  • Computational Complexity: The paper notes that the simplicity of message-passing operations might lead to slightly inferior performance compared to more complex belief propagation decoding, though it still outperforms prior methods.
  • Parameter Tuning: The theoretical analysis suggests that very large coupling parameters (Ω) might be required for exact thresholds, but simulations indicate that moderate values of Ω (e.g., 2-3) still yield excellent results, suggesting a practical sweet spot.
  • Future Work: The authors plan to assess the learning phase performance on natural image databases and explore the architecture's capabilities in real-world scenarios.

Summary updated

Abstract

We propose a novel architecture to design a neural associative memory that is capable of learning a large number of patterns and recalling them later in presence of noise. It is based on dividing the neurons into local clusters and parallel plains, an architecture that is similar to the visual cortex of macaque brain. The common features of our proposed model with those of spatially-coupled codes enable us to show that the performance of such networks in eliminating noise is drastically better than the previous approaches while maintaining the ability of learning an exponentially large number of patterns. Previous work either failed in providing good performance during the recall phase or in offering large pattern retrieval (storage) capacities. We also present computational experiments that lend additional support to the theoretical analysis.
BibTeX
@inproceedings{karbasi2013coupled,
  author = {Amin Karbasi and Amir Hesam Salavati and Amin Shokrollahi},
  title = {{Coupled neural associative memories.}},
  booktitle = {ITW},
  year = {2013},
  url = {https://openreview.net/forum?id=NI0UKhgxzBM4},
}
Figure