Variational Memory Addressing in Generative Models

Jörg Bornschein, Andriy Mnih, Daniel Zoran, Danilo Jimenez Rezende

Introduction

Augmenting generative models, such as Variational Autoencoders (VAEs), with external memory has traditionally relied on soft attention mechanisms. In soft attention, the model retrieves a weighted average of all items in memory. While this approach is easily trained using standard backpropagation, it suffers from two major drawbacks. First, averaging multiple memory slots often leads to "blurry" representations, which is highly undesirable in generative tasks where distinct objects or styles are required. Second, as the memory size scales, computing a weight for every single slot becomes computationally expensive.

To overcome these limitations, this paper introduces a method for stochastic addressing—essentially a "hard" attention mechanism where the model selects one specific memory slot. By treating memory addresses as discrete latent variables, the model learns to pick specific "templates" from memory and uses a continuous latent variable to refine those templates into final images. This approach naturally encourages multimodality, making it particularly well-suited for high-dimensional data like images, and allows the model to scale to much larger memories than standard soft attention methods.

Methodology: Stochastic Addressing and VIMCO

The core architecture is a VAE augmented with an external memory buffer $M$. Instead of a weighted sum, the model uses a discrete latent variable $a$ to represent the specific address in memory, sampling it from a categorical distribution.

Graphical model illustrating the generative process with a discrete address variable $a$, memory $m$, and continuous latent $z$.

The Generative Process: To generate an observation $x$ (e.g., an image), the model first picks an address $a$ and retrieves the corresponding content $m_a$ (acting as a "template"). It then uses a continuous latent variable $z$ to model the specific variations—such as distortions, rotations, or stroke changes—needed to transform that template into the final output.

Variational Inference: Because the address $a$ is a discrete choice, standard backpropagation cannot be used directly. To solve this, the authors employ VIMCO, a multi-sample variational objective. VIMCO allows the model to estimate gradients for the discrete addressing logic by comparing multiple samples from the posterior. To decide which memory slot to pick, the model learns a similarity measure based on the inner product of embeddings, normalized by the norm of the memory content.

Key Insights

Treating memory-read operations as a conditional mixture distribution offers several theoretical and practical advantages:

  • Separation of Concerns: By utilizing both a discrete address $a$ and a continuous variable $z$, the model cleanly separates identity (e.g., "which character am I drawing?") from variation (e.g., "how is this specific instance drawn?").
  • KL Divergence as a Metric: Treating the chosen address as a latent variable allows the use of KL divergence over $a$ to quantify exactly how much information the model is extracting from its memory during the generative process.
  • Non-parametric Flexibility: Because the memory is external, its size or content can be modified at test time. For instance, hundreds of new, unseen characters can be added without retraining the neural network's parameters.

Key Results and Performance

The authors evaluated their model on Binarized MNIST (for "train to recall" tasks) and Omniglot. Omniglot, featuring thousands of characters with only 20 examples each, serves as a gold standard for generative few-shot learning.

Training dynamics showing NLL and KL divergence estimates, alongside generated image samples.
  • Generative Few-Shot Learning: On the Omniglot dataset, the variational memory model achieved a Negative Log-Likelihood (NLL) of ~68.8 nats, significantly outperforming a standard unconditioned VAE, which scored ~90.8 nats.
  • Scaling Robustness and Efficiency: Unlike soft-attention baselines—which suffer sharp performance drops when "distractor" items are added—the stochastic addressing model remains stable. It successfully identifies relevant memory content even with hundreds of unseen Omniglot characters loaded into memory. Furthermore, at scale (memory size $|M| \geq 48$), this "hard" addressing becomes faster in wall-clock time per update than soft-attention alternatives.
  • Classification: Though primarily designed for generation, the model proved highly capable in classification tasks, achieving 91% accuracy on 5-way 1-shot and 77% on 20-way 1-shot Omniglot tasks without any task-specific fine-tuning.

Limitations and Future Work

While highly effective, the approach has a few limitations. The authors note that if the memory buffer is too large during the initial phase of training (e.g., $|M| > 512$), the model can sometimes struggle to converge to an optimal solution. Additionally, VIMCO requires multiple samples per training step, increasing the computational overhead per iteration compared to a simple VAE. Finally, in these few-shot experiments, the memory was populated by raw pixels; future work could explore more complex "write" operations or learned, abstract memory representations to further enhance performance.

Summary updated

Abstract

Aiming to augment generative models with external memory, we interpret the output of a memory module with stochastic addressing as a conditional mixture distribution, where a read operation corresponds to sampling a discrete memory address and retrieving the corresponding content from memory. This perspective allows us to apply variational inference to memory addressing, which enables effective training of the memory module by using the target information to guide memory lookups. Stochastic addressing is particularly well-suited for generative models as it naturally encourages multimodality which is a prominent aspect of most high-dimensional datasets. Treating the chosen address as a latent variable also allows us to quantify the amount of information gained with a memory lookup and measure the contribution of the memory module to the generative process. To illustrate the advantages of this approach we incorporate it into a variational autoencoder and apply the resulting model to the task of generative few-shot learning. The intuition behind this architecture is that the memory module can pick a relevant template from memory and the continuous part of the model can concentrate on modeling remaining variations. We demonstrate empirically that our model is able to identify and access the relevant memory contents even with hundreds of unseen Omniglot characters in memory.
BibTeX
@inproceedings{bornschein2017variational,
  author = {Jörg Bornschein and Andriy Mnih and Daniel Zoran and Danilo Jimenez Rezende},
  title = {{Variational Memory Addressing in Generative Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=rkZ9H_-_bS},
}
Figure