Introduction
While deep learning has mastered many domains, robust and efficient "memory" remains a persistent challenge. Standard Recurrent Neural Networks (RNNs) and LSTMs rely on "active" memory, keeping information alive through continuous neuronal activity. This approach is energy-expensive and prone to failure over long delays. Conversely, Memory-Augmented Neural Networks (like the Neural Turing Machine) often use a "digital-like" differentiable memory that, while powerful, lacks biological plausibility.
H-Mem is a novel neural network architecture designed to bridge this gap. By utilizing biologically inspired Hebbian plasticity—the principle that "cells that fire together, wire together"—H-Mem creates a memory system that stores and retrieves associations over medium-to-long time scales. By integrating a hetero-associative memory module that updates its own weights "on the fly," the model achieves one-shot learning and solves complex reasoning tasks that typically baffle standard recurrent networks.
The H-Mem Architecture
The H-Mem model is built around a core hetero-associative network subject to Hebbian plasticity. The architecture consists of three primary components:
- Input Encoder (IE): Converts raw inputs, such as images or sentences, into a dense vector embedding.
- Store-Branch: When the network encounters a "fact," this branch generates a key-vector and a value-vector. These vectors are used to update a central associative memory matrix ($W^{assoc}$) using a specific Hebbian learning rule. Crucially, this matrix is not learned via backpropagation; it changes dynamically during inference based on the input.
- Recall-Branch: When the network receives a "query," this branch generates a query key to probe the memory matrix and retrieve the associated value.
The Hebbian Rule: The authors employ a modified version of Oja’s rule for the memory matrix. It includes a "soft" upper bound to prevent weights from exploding and a decay term that helps the network prioritize new associations over old, irrelevant ones.
For tasks requiring multi-step reasoning, the authors also developed a recurrent version of the recall branch. This allows the network to query its own memory multiple times, chaining facts together to deduce an answer (e.g., "Where is the football?" $\rightarrow$ "John has it" $\rightarrow$ "Where is John?").
Key Insights and Performance
The model was evaluated on two distinct challenges: a custom Single-Shot Association Task (pairing MNIST digits with CIFAR-10 objects across long delays) and the bAbI Tasks (a suite of 20 synthetic story tasks designed to test reasoning).
Learning to Use Plasticity
The most profound insight from H-Mem is that while the memory matrix uses a fixed Hebbian rule, the controllers (the store and recall branches) are trained via gradient descent. The network essentially "learns how to remember." Unlike LSTMs, which need many examples to learn a relationship, H-Mem can bind two arbitrary concepts in a single time step.
Robustness to Delay
Because H-Mem stores memory in "weights" (synapses) rather than "activity," it is remarkably resistant to interference. In the single-shot association task, H-Mem maintained approximately 90% accuracy regardless of the delay length. In stark contrast, LSTM performance plummeted to near-chance levels (~10%) after just 20 steps of delay.
bAbI Mastery and Data Efficiency
The extended H-Mem model solved 20 out of 20 bAbI tasks with a mean error of only 0.6%. This significantly outperforms standard LSTMs (which failed 16/20 tasks) and is highly competitive with state-of-the-art non-biological models like EntNet and MemN2N. Furthermore, when trained on a restricted dataset of only 1,000 examples, H-Mem significantly outperformed the Entity Network (EntNet), suggesting that Hebbian structures are highly sample-efficient.
Limitations and Future Work
While H-Mem takes a strong step toward biologically plausible memory, limitations remain. The inference and memory update mechanisms are biologically inspired, but the training of the controllers still relies on backpropagation. Additionally, the current model uses simplified rate-based neurons. Transitioning to spiking neural networks is a logical next step, potentially allowing H-Mem to run efficiently on ultra-low-power neuromorphic hardware. Finally, as the bAbI tasks are synthetic with small vocabularies, future work will need to test how well H-Mem scales to massive, real-world natural language processing datasets.
Summary updated