Abstract
Aiming to augment generative models with external memory, we interpret the output of a memory module with stochastic addressing as a conditional mixture distribution, where a read operation corresponds to sampling a discrete memory address and retrieving the corresponding content from memory. This perspective allows us to apply variational inference to memory addressing, which enables effective training of the memory module by using the target information to guide memory lookups. Stochastic addressing is particularly well-suited for generative models as it naturally encourages multimodality which is a prominent aspect of most high-dimensional datasets. Treating the chosen address as a latent variable also allows us to quantify the amount of information gained with a memory lookup and measure the contribution of the memory module to the generative process. To illustrate the advantages of this approach we incorporate it into a variational autoencoder and apply the resulting model to the task of generative few-shot learning. The intuition behind this architecture is that the memory module can pick a relevant template from memory and the continuous part of the model can concentrate on modeling remaining variations. We demonstrate empirically that our model is able to identify and access the relevant memory contents even with hundreds of unseen Omniglot characters in memory.
Introduction
Deep learning models have become increasingly proficient at generating high-dimensional data. However, they often struggle with few-shot learning—the ability to generate variations of a concept after seeing only a single example. While external memory buffers have been proposed to address this, standard "soft attention" mechanisms face scaling and conceptual limitations. Soft attention calculates a weighted average across all memory slots, which is computationally expensive for large memories and often results in "blurry" mixtures of templates rather than precise retrievals.
This paper introduces Variational Memory Addressing, a method that treats memory retrieval as a stochastic process using a discrete latent variable. By picking a single memory slot instead of averaging over many, the model scales gracefully and maintains the precision necessary for high-quality generative tasks.
Methods: Stochastic Addressing and Variational Inference
The core innovation lies in treating the memory address as a discrete latent variable $a$. In the generative process, the model first samples an address $a$ from a prior distribution and retrieves the corresponding content $m_a$ from the memory buffer. It then samples a continuous latent variable $z$, representing style or local variations, and combines it with $m_a$ to produce the final output $x$.
To train this model, the authors employ variational inference. Since discrete choices are non-differentiable, they use the VIMCO multi-sample variational objective to estimate gradients for the addressing logic. The model learns a similarity function to embed both inputs and memory contents into a shared space, using a normalized inner product to determine the "match" between a query and a memory entry. This architecture effectively separates the "what" (the template from memory) from the "how" (the continuous variations handled by the VAE).
Experiments and Datasets
The model was evaluated on two primary benchmarks:
- Binarized MNIST: Used to test "train to recall" capabilities, determining if the model could learn to perfectly retrieve specific digits from a learned memory.
- Omniglot: A dataset of 1,623 handwritten characters from 50 alphabets, serving as the benchmark for few-shot learning. Here, the model must generalize to entirely unseen character classes.
Key Results and Insights
The stochastic addressing approach yielded several significant advantages:
- Scalability and Precision: The model successfully handled memories with up to 2,048 items. Unlike soft-attention models, which degrade as irrelevant "confounder" items are added, this model remains robust by learning to ignore irrelevant slots.
- Information Gain: By measuring the KL divergence of the address variable $a$, the authors quantified the information gained from memory lookups. High KL values indicate confident, specific template selection.
- Few-Shot Performance: On Omniglot, the model significantly outperformed soft-attention baselines. It maintained high performance even when the memory contained hundreds of different character classes.
Furthermore, the model achieved 90% accuracy on 20-way 5-shot classification on Omniglot, despite being a purely generative model without explicit discriminative training.
Conclusion and Future Work
Variational Memory Addressing provides a principled way to integrate large-scale external memory into generative models. By treating memory as a conditional mixture distribution, the model can grow its knowledge base without retraining. While the current work focuses on reading from static or learned memories, future research could explore dynamic "writing" mechanisms and more complex memory structures to further enhance few-shot learning capabilities.
Summary updated