Meta-Learning Deep Energy-Based Memory Models

Sergey Bartunov, Jack W. Rae, Simon Osindero, Timothy P. Lillicrap

Abstract

We study the problem of learning associative memory—a system which is able to retrieve a remembered pattern based on its distorted or incomplete version. Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by running the dynamics starting from a query pattern until it converges to an attractor. In such models, the dynamics are often implemented as an optimization procedure that minimizes an energy function, such as in the classical Hopfield network. In general, it is difficult to derive a writing rule for a given dynamics and energy that is both compressive and fast. Thus, most research in energy-based memory has been limited either to tractable energy models not expressive enough to handle complex high-dimensional objects such as natural images, or to models that do not offer fast writing. We present a novel meta-learning approach to energy-based memory models (EBMM) that allows one to use an arbitrary neural architecture as an energy model and quickly store patterns in its weights. We demonstrate experimentally that our EBMM approach can build compressed memories for synthetic and natural data, and is capable of associative retrieval that outperforms existing memory systems in terms of reconstruction error and compression rate.

Introduction

Associative memory is a fundamental cognitive capability that allows a system to retrieve a complete, stored pattern from a noisy or partial query. Historically, this has been modeled using attractor networks, where memories are represented as local minima in an energy landscape. To retrieve a memory, the system performs gradient descent on the energy function, starting from the query point and "sliding down" to the nearest attractor.

However, a persistent challenge in this field is the trade-off between expressivity and speed. Classic models like the Hopfield network are fast to "write" to but lack the capacity to handle high-dimensional data like natural images. Conversely, deep generative models (like VAEs or RBMs) are highly expressive but require thousands of gradient descent iterations to store new patterns, making them impractical for fast adaptation. This paper introduces Energy-Based Memory Models (EBMM), which bridge this gap by using meta-learning to enable fast, expressive, and compressive memory storage.

The Energy-Based Memory Model (EBMM)

The EBMM architecture uses a deep neural network to map input x to a scalar energy value E.

The core of the EBMM is a deep neural network (e.g., a ResNet) that functions as an energy model. This network takes an input $x$ and outputs a single scalar value $E(x; \theta)$, representing the "energy" of that state.

Fast Reading (Retrieval)

Retrieval is performed by taking a small number of gradient descent steps (typically 5) on the input pixels to minimize the energy. If the energy landscape is correctly shaped, a noisy query will naturally converge to the nearest stored memory.

Fast Writing (Storage)

Instead of using a fixed, hand-coded writing rule, EBMM uses gradient descent on the network's weights $\theta$. The goal is to modify the weights such that the energy landscape is "carved" to have local minima at the locations of the patterns we wish to remember.

Meta-Learning the Writing Rule

The complete EBMM pipeline: A meta-learned initialization allows a few gradient steps to transform a flat energy landscape into one with specific attractors for stored patterns.

The primary innovation of this work is the application of gradient-based meta-learning (similar to MAML) to the writing process. The model meta-trains the initial weights $\bar{\theta}$ and the learning rate schedules so that the network is "primed" for memory storage.

As shown in Figure 4, the process begins with a relatively flat energy landscape. Through the meta-learned write operation—which involves just a few steps of gradient descent on the weights—the landscape is reshaped to create deep basins of attraction at the target coordinates $x_1, x_2, x_3$. To ensure stability, the writing loss includes both the energy value and the norm of the gradient, forcing the gradient to zero at the stored patterns.

Experimental Evaluation

The authors evaluated EBMM against several baselines, including LSTMs, Memory-Augmented Neural Networks (MANN), and Hopfield networks, across several challenging datasets:

  • Omniglot & MNIST: EBMM demonstrated excellent fast adaptation. Notably, a model meta-trained on Omniglot generalized to MNIST characters without further training, indicating it had learned a general strategy for remembering shapes.
  • CIFAR-10 & ImageNet 64x64: On these high-dimensional natural image datasets, EBMM significantly outperformed baselines in reconstruction error and compression rate. It successfully stored and retrieved complex images using only 5 iterations for both writing and reading.
  • Capacity Testing: Using random binary strings, the authors confirmed that EBMM offers high information capacity, utilizing the network's weights as distributed storage rather than relying on fixed "memory slots."

Key Insights

  1. Weights as Distributed Storage: By using weights instead of slots, EBMM learns a "prior" about the data distribution. It understands the underlying structure of images (edges, textures), allowing it to compress new memories more efficiently than models that treat every input as an independent vector.
  2. Energy as Familiarity: The energy levels in EBMM are naturally interpretable. Stored memories have the lowest energy, while distorted or out-of-distribution images have higher energy, making the model a candidate for anomaly detection.
  3. Avoiding Spurious Attractors: By unrolling the optimization loop during training, the meta-learning process explicitly teaches the model to avoid creating "false memories" or spurious attractors that often plague simpler energy-based models.

Conclusion and Future Work

EBMM represents a significant step forward in associative memory, providing a deep, expressive system that can be updated rapidly. While the current model assumes batch writing and can be computationally intensive to train due to unrolled optimization, it sets a new standard for reconstruction quality and compression in memory systems. Future work will focus on adapting EBMM to online settings to mitigate catastrophic forgetting and exploring perceptual losses to improve the visual fidelity of retrieved high-resolution images.

Summary updated

Abstract

We study the problem of learning associative memory -- a system which is able to retrieve a remembered pattern based on its distorted or incomplete version. Attractor networks provide a sound model of associative memory: patterns are stored as attractors of the network dynamics and associative retrieval is performed by running the dynamics starting from a query pattern until it converges to an attractor. In such models the dynamics are often implemented as an optimization procedure that minimizes an energy function, such as in the classical Hopfield network. In general it is difficult to derive a writing rule for a given dynamics and energy that is both compressive and fast. Thus, most research in energy-based memory has been limited either to tractable energy models not expressive enough to handle complex high-dimensional objects such as natural images, or to models that do not offer fast writing. We present a novel meta-learning approach to energy-based memory models (EBMM) that allows one to use an arbitrary neural architecture as an energy model and quickly store patterns in its weights. We demonstrate experimentally that our EBMM approach can build compressed memories for synthetic and natural data, and is capable of associative retrieval that outperforms existing memory systems in terms of the reconstruction error and compression rate.
BibTeX
@inproceedings{bartunov2020meta,
  author = {Sergey Bartunov and Jack W. Rae and Simon Osindero and Timothy P. Lillicrap},
  title = {{Meta-Learning Deep Energy-Based Memory Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=uCAIou9CXCM},
}
Figure