Introduction
Augmenting generative models, such as Variational Autoencoders (VAEs), with external memory has traditionally relied on soft attention mechanisms. In soft attention, the model retrieves a weighted average of all items in memory. While this approach is easily trained using standard backpropagation, it suffers from two major drawbacks. First, averaging multiple memory slots often leads to "blurry" representations, which is highly undesirable in generative tasks where distinct objects or styles are required. Second, as the memory size scales, computing a weight for every single slot becomes computationally expensive.
To overcome these limitations, this paper introduces a method for stochastic addressing—essentially a "hard" attention mechanism where the model selects one specific memory slot. By treating memory addresses as discrete latent variables, the model learns to pick specific "templates" from memory and uses a continuous latent variable to refine those templates into final images. This approach naturally encourages multimodality, making it particularly well-suited for high-dimensional data like images, and allows the model to scale to much larger memories than standard soft attention methods.
Methodology: Stochastic Addressing and VIMCO
The core architecture is a VAE augmented with an external memory buffer $M$. Instead of a weighted sum, the model uses a discrete latent variable $a$ to represent the specific address in memory, sampling it from a categorical distribution.
The Generative Process: To generate an observation $x$ (e.g., an image), the model first picks an address $a$ and retrieves the corresponding content $m_a$ (acting as a "template"). It then uses a continuous latent variable $z$ to model the specific variations—such as distortions, rotations, or stroke changes—needed to transform that template into the final output.
Variational Inference: Because the address $a$ is a discrete choice, standard backpropagation cannot be used directly. To solve this, the authors employ VIMCO, a multi-sample variational objective. VIMCO allows the model to estimate gradients for the discrete addressing logic by comparing multiple samples from the posterior. To decide which memory slot to pick, the model learns a similarity measure based on the inner product of embeddings, normalized by the norm of the memory content.
Key Insights
Treating memory-read operations as a conditional mixture distribution offers several theoretical and practical advantages:
- Separation of Concerns: By utilizing both a discrete address $a$ and a continuous variable $z$, the model cleanly separates identity (e.g., "which character am I drawing?") from variation (e.g., "how is this specific instance drawn?").
- KL Divergence as a Metric: Treating the chosen address as a latent variable allows the use of KL divergence over $a$ to quantify exactly how much information the model is extracting from its memory during the generative process.
- Non-parametric Flexibility: Because the memory is external, its size or content can be modified at test time. For instance, hundreds of new, unseen characters can be added without retraining the neural network's parameters.
Key Results and Performance
The authors evaluated their model on Binarized MNIST (for "train to recall" tasks) and Omniglot. Omniglot, featuring thousands of characters with only 20 examples each, serves as a gold standard for generative few-shot learning.
- Generative Few-Shot Learning: On the Omniglot dataset, the variational memory model achieved a Negative Log-Likelihood (NLL) of ~68.8 nats, significantly outperforming a standard unconditioned VAE, which scored ~90.8 nats.
- Scaling Robustness and Efficiency: Unlike soft-attention baselines—which suffer sharp performance drops when "distractor" items are added—the stochastic addressing model remains stable. It successfully identifies relevant memory content even with hundreds of unseen Omniglot characters loaded into memory. Furthermore, at scale (memory size $|M| \geq 48$), this "hard" addressing becomes faster in wall-clock time per update than soft-attention alternatives.
- Classification: Though primarily designed for generation, the model proved highly capable in classification tasks, achieving 91% accuracy on 5-way 1-shot and 77% on 20-way 1-shot Omniglot tasks without any task-specific fine-tuning.
Limitations and Future Work
While highly effective, the approach has a few limitations. The authors note that if the memory buffer is too large during the initial phase of training (e.g., $|M| > 512$), the model can sometimes struggle to converge to an optimal solution. Additionally, VIMCO requires multiple samples per training step, increasing the computational overhead per iteration compared to a simple VAE. Finally, in these few-shot experiments, the memory was populated by raw pixels; future work could explore more complex "write" operations or learned, abstract memory representations to further enhance performance.
Summary updated