Abstract
Despite recent advances, the remaining bottlenecks in deep generative models are the necessity of extensive training and difficulties with generalization from a small number of training examples. We develop a new framework for few-shot generative modelling, which is able to learn a generative model of a new concept from a few examples. Our approach, Generative Matching Networks (GMNs), is based on the idea of matching networks for supervised learning but is adapted to the unsupervised setting.
Introduction
Standard deep generative models, such as Variational Autoencoders (VAEs) or Generative Adversarial Networks (GANs), are notoriously data-hungry. They typically require thousands of examples to learn a concept and suffer from "catastrophic interference"—if you try to teach a trained model a new concept, it often forgets what it previously learned unless you retrain it on everything.
In contrast, humans can see a single instance of a new object and immediately imagine variations of it. This paper tackles the "few-shot generative modeling" challenge: how can a model instantly adapt its generative distribution to a new domain or class using only a handful of examples ($1$ to $20$) provided at test time?
Methods
The authors introduce Generative Matching Networks (GMNs). The core innovation is the integration of a non-parametric attentional mechanism into a probabilistic framework.
As shown in Figure 1, the architecture operates in three primary stages:
- The Matching Mechanism: Instead of generating an image solely from a latent vector $z$, the model looks at a "conditioning set" of images $X$. It projects both the latent vector and the conditioning images into a shared "matching space." It then calculates an attention kernel (similarity) to determine which conditioning images are most relevant.
- Prototype Aggregation: The model extracts features from the relevant conditioning images and blends them into a "prototype" vector $r_G$. This vector, along with the latent $z$, is passed to a decoder to produce the final image.
- Full Context Matching & Data-Dependent Prior: To help the model understand the relationship between images in the conditioning set, a recurrent controller makes multiple passes over the data. Unlike a standard VAE, GMNs adjust their prior distribution based on the conditioning set, becoming more "certain" as more data becomes available.
Key Results
The primary benchmark used is Omniglot, containing 1,623 different handwritten characters. GMNs significantly outperformed VAEs and the "Neural Statistician" in conditional log-likelihood. On Omniglot, as the number of conditioning examples increased from 1 to 19, the GMN's predictive performance improved much more sharply than its competitors.
Qualitatively, the model produces sharp, recognizable variations of characters it has never seen during training (Figure 6). The authors found that "pseudo-inputs"—trainable parameters acting as a default template—prevent the model from overfitting to the conditioning set. Even though GMNs are generative, they can be used for classification; in a 5-class, 5-shot setting, GMNs achieved 97.4% accuracy, rivaling supervised models.
Limitations and Future Work
While GMNs excel on character strokes, it remains to be seen how well this attentional matching scales to high-resolution, complex natural images. Additionally, "Full Context Matching" involves multiple passes with an RNN, which increases computational overhead. Future work could explore applying GMNs to semi-supervised learning or extending the architecture to handle more complex, hierarchical data structures.
Summary updated