Variational Memory Addressing in Generative Models

Jörg Bornschein, Andriy Mnih, Daniel Zoran, Danilo Jimenez Rezende

Abstract

Aiming to augment generative models with external memory, we interpret the output of a memory module with stochastic addressing as a conditional mixture distribution, where a read operation corresponds to sampling a discrete memory address and retrieving the corresponding content from memory. This perspective allows us to apply variational inference to memory addressing, which enables effective training of the memory module by using the target information to guide memory lookups. Stochastic addressing is particularly well-suited for generative models as it naturally encourages multimodality which is a prominent aspect of most high-dimensional datasets. Treating the chosen address as a latent variable also allows us to quantify the amount of information gained with a memory lookup and measure the contribution of the memory module to the generative process. To illustrate the advantages of this approach we incorporate it into a variational autoencoder and apply the resulting model to the task of generative few-shot learning. The intuition behind this architecture is that the memory module can pick a relevant template from memory and the continuous part of the model can concentrate on modeling remaining variations. We demonstrate empirically that our model is able to identify and access the relevant memory contents even with hundreds of unseen Omniglot characters in memory.

Introduction

Deep learning models have become increasingly proficient at generating high-dimensional data. However, they often struggle with few-shot learning—the ability to generate variations of a concept after seeing only a single example. While external memory buffers have been proposed to address this, standard "soft attention" mechanisms face scaling and conceptual limitations. Soft attention calculates a weighted average across all memory slots, which is computationally expensive for large memories and often results in "blurry" mixtures of templates rather than precise retrievals.

This paper introduces Variational Memory Addressing, a method that treats memory retrieval as a stochastic process using a discrete latent variable. By picking a single memory slot instead of averaging over many, the model scales gracefully and maintains the precision necessary for high-quality generative tasks.

Methods: Stochastic Addressing and Variational Inference

The core innovation lies in treating the memory address as a discrete latent variable $a$. In the generative process, the model first samples an address $a$ from a prior distribution and retrieves the corresponding content $m_a$ from the memory buffer. It then samples a continuous latent variable $z$, representing style or local variations, and combines it with $m_a$ to produce the final output $x$.

Graphical model of the proposed generative process with stochastic memory addressing.

To train this model, the authors employ variational inference. Since discrete choices are non-differentiable, they use the VIMCO multi-sample variational objective to estimate gradients for the addressing logic. The model learns a similarity function to embed both inputs and memory contents into a shared space, using a normalized inner product to determine the "match" between a query and a memory entry. This architecture effectively separates the "what" (the template from memory) from the "how" (the continuous variations handled by the VAE).

Experiments and Datasets

The model was evaluated on two primary benchmarks:

  1. Binarized MNIST: Used to test "train to recall" capabilities, determining if the model could learn to perfectly retrieve specific digits from a learned memory.
  2. Omniglot: A dataset of 1,623 handwritten characters from 50 alphabets, serving as the benchmark for few-shot learning. Here, the model must generalize to entirely unseen character classes.

Key Results and Insights

The stochastic addressing approach yielded several significant advantages:

  • Scalability and Precision: The model successfully handled memories with up to 2,048 items. Unlike soft-attention models, which degrade as irrelevant "confounder" items are added, this model remains robust by learning to ignore irrelevant slots.
  • Information Gain: By measuring the KL divergence of the address variable $a$, the authors quantified the information gained from memory lookups. High KL values indicate confident, specific template selection.
  • Few-Shot Performance: On Omniglot, the model significantly outperformed soft-attention baselines. It maintained high performance even when the memory contained hundreds of different character classes.
Generated character samples demonstrating the model's ability to produce variations of templates.

Furthermore, the model achieved 90% accuracy on 20-way 5-shot classification on Omniglot, despite being a purely generative model without explicit discriminative training.

Conclusion and Future Work

Variational Memory Addressing provides a principled way to integrate large-scale external memory into generative models. By treating memory as a conditional mixture distribution, the model can grow its knowledge base without retraining. While the current work focuses on reading from static or learned memories, future research could explore dynamic "writing" mechanisms and more complex memory structures to further enhance few-shot learning capabilities.

Summary updated

Abstract

Aiming to augment generative models with external memory, we interpret the output of a memory module with stochastic addressing as a conditional mixture distribution, where a read operation corresponds to sampling a discrete memory address and retrieving the corresponding content from memory. This perspective allows us to apply variational inference to memory addressing, which enables effective training of the memory module by using the target information to guide memory lookups. Stochastic addressing is particularly well-suited for generative models as it naturally encourages multimodality which is a prominent aspect of most high-dimensional datasets. Treating the chosen address as a latent variable also allows us to quantify the amount of information gained with a memory lookup and measure the contribution of the memory module to the generative process. To illustrate the advantages of this approach we incorporate it into a variational autoencoder and apply the resulting model to the task of generative few-shot learning. The intuition behind this architecture is that the memory module can pick a relevant template from memory and the continuous part of the model can concentrate on modeling remaining variations. We demonstrate empirically that our model is able to identify and access the relevant memory contents even with hundreds of unseen Omniglot characters in memory
BibTeX
@inproceedings{bornschein2017variational,
  author = {Jörg Bornschein and Andriy Mnih and Daniel Zoran and Danilo Jimenez Rezende},
  title = {{Variational Memory Addressing in Generative Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=rkZ9H_-_bS},
}

Loading related content...

Figure