Meta-Learning Deep Energy-Based Memory Models

Sergey Bartunov, Jack W. Rae, Simon Osindero, Timothy P. Lillicrap

Introduction

Associative memory—the ability to retrieve a complete, stored pattern when presented with a distorted or partial version—is a fundamental cognitive function. Historically, this "mental autocomplete" was modeled using attractor networks like the Hopfield network. While elegant, classical models suffer from low storage capacity and struggle with complex, high-dimensional data like natural images. Conversely, modern deep learning alternatives like Variational Auto-encoders (VAEs) are highly expressive but notoriously "slow" to learn, often requiring thousands of gradient updates to store a new pattern.

Researchers from DeepMind have developed a "best of both worlds" solution: Meta-Learning Deep Energy-Based Memory Models (EBMM). By using meta-learning to optimize the "writing" process, EBMM turns the weights of a deep neural network into a high-capacity associative memory. It can store complex patterns in just five gradient steps and retrieve them instantly, significantly outperforming traditional architectures in capacity and reconstruction quality.

The EBMM Approach

The authors propose treating a neural network as an energy function, $E(x; \theta)$. In this framework, "reading" from memory is an optimization process where an input $x$ moves down the energy gradient until it hits a local minimum, or attractor.

The EBMM process: Initializing a flat energy landscape, writing patterns to create attractors, and reading by descending the gradient.

The core innovation is the Meta-Learning Writing Rule:

  • Fast Writing: Instead of traditional training, EBMM uses a meta-learning approach (inspired by MAML) to find an optimal initial set of weights and a learning rate schedule. This allows the model to store a batch of new images in just 5 gradient steps.
  • The Writing Loss: To ensure a pattern becomes a stable memory, the writing rule minimizes a three-part loss function:
  • Energy: Lowers the energy value at the target pattern's location.
  • Gradient Norm: Ensures the gradient at that location is zero, making it a stable stationary point.
  • Weight Anchor: Prevents the weights from drifting too far from their meta-learned initialization.
  • End-to-End Training: The entire system—initial weights, writing steps, and reading steps—is trained end-to-end to minimize the final reconstruction error.

Key Insights

Unlike "slot-based" memories that store data in a matrix, EBMM stores data in the functional relationships of the network weights, allowing for massive compression rates. The paper treats the unrolled steps of gradient descent as layers in a meta-network, effectively learning an "optimization schedule" specifically tuned for memory storage and retrieval.

Interestingly, the model learns to assign lower energy to "remembered" patterns and higher energy to unknown or distorted patterns. This allows it to act as an out-of-distribution detector without explicit training. Furthermore, by including the gradient norm in the writing loss, the model ensures that the "basins of attraction" are sharp and stable, preventing retrieved images from being blurry.

Results and Performance

The model was rigorously evaluated across several benchmarks, including Omniglot, CIFAR-10, 64x64 ImageNet, and random binary strings.

Iterative retrieval of Omniglot characters from noisy queries, showing the corresponding drop in energy.
  • Superior Compression: On CIFAR-10 and ImageNet, EBMM significantly outperformed Memory-Augmented Neural Networks (MANN) and Memory Networks, achieving lower reconstruction errors using the same number of parameters.
  • Natural Image Mastery: While classical Hopfield networks fail on natural images, EBMM successfully retrieved recognizable 64x64 ImageNet photos even when 25% of the query image was occluded.
  • Zero-Shot Transfer: A model trained on Omniglot characters successfully retrieved MNIST digits, proving that the learned "writing" strategy generalizes to similar but unseen data distributions.

Limitations and Future Work

While highly effective, the current EBMM assumes a fixed batch of patterns is written at once. Moving to an online or sequential writing mode is a necessary next step for integration into reinforcement learning agents. Additionally, meta-learning through unrolled optimization is computationally expensive, currently limiting the network size. Scaling to full-resolution ImageNet will likely require advanced techniques like implicit differentiation or synthetic gradients to overcome the bottleneck of recurrent training.

Summary updated

Abstract

We study the problem of learning an associative memory model -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version. Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by running the dynamics starting from a query pattern until it converges to an attractor. In such models the dynamics are often implemented as an optimization procedure that minimizes an energy function, such as in the classical Hopfield network. In general it is difficult to derive a writing rule for a given dynamics and energy that is both compressive and fast. Thus, most research in energy-based memory has been limited either to tractable energy models not expressive enough to handle complex high-dimensional objects such as natural images, or to models that do not offer fast writing. We present a novel meta-learning approach to energy-based memory models (EBMM) that allows one to use an arbitrary neural architecture as an energy model and quickly store patterns in its weights. We demonstrate experimentally that our EBMM approach can build compressed memories for synthetic and natural data, and is capable of associative retrieval that outperforms existing memory systems in terms of the reconstruction error and compression rate.
BibTeX
@inproceedings{bartunov2020meta,
  author = {Sergey Bartunov and Jack W. Rae and Simon Osindero and Timothy P. Lillicrap},
  title = {{Meta-Learning Deep Energy-Based Memory Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=uCAIou9CXCM},
}
Figure