Few-shot Generative Modelling with Generative Matching Networks

Sergey Bartunov, Dmitry P. VetrovAISTATS 2018 · OpenReview

Abstract

Despite recent advances, the remaining bottlenecks in deep generative models are the necessity of extensive training and difficulties with generalization from a small number of training examples. We develop a new framework for few-shot generative modelling, which is able to learn a generative model of a new concept from a few examples. Our approach, Generative Matching Networks (GMNs), is based on the idea of matching networks for supervised learning but is adapted to the unsupervised setting.

Introduction

Standard deep generative models, such as Variational Autoencoders (VAEs) or Generative Adversarial Networks (GANs), are notoriously data-hungry. They typically require thousands of examples to learn a concept and suffer from "catastrophic interference"—if you try to teach a trained model a new concept, it often forgets what it previously learned unless you retrain it on everything.

In contrast, humans can see a single instance of a new object and immediately imagine variations of it. This paper tackles the "few-shot generative modeling" challenge: how can a model instantly adapt its generative distribution to a new domain or class using only a handful of examples ($1$ to $20$) provided at test time?

Methods

The authors introduce Generative Matching Networks (GMNs). The core innovation is the integration of a non-parametric attentional mechanism into a probabilistic framework.

The three-stage process of GMNs: computing attention, aggregating prototypes, and decoding.

As shown in Figure 1, the architecture operates in three primary stages:

  1. The Matching Mechanism: Instead of generating an image solely from a latent vector $z$, the model looks at a "conditioning set" of images $X$. It projects both the latent vector and the conditioning images into a shared "matching space." It then calculates an attention kernel (similarity) to determine which conditioning images are most relevant.
  2. Prototype Aggregation: The model extracts features from the relevant conditioning images and blends them into a "prototype" vector $r_G$. This vector, along with the latent $z$, is passed to a decoder to produce the final image.
  3. Full Context Matching & Data-Dependent Prior: To help the model understand the relationship between images in the conditioning set, a recurrent controller makes multiple passes over the data. Unlike a standard VAE, GMNs adjust their prior distribution based on the conditioning set, becoming more "certain" as more data becomes available.

Key Results

The primary benchmark used is Omniglot, containing 1,623 different handwritten characters. GMNs significantly outperformed VAEs and the "Neural Statistician" in conditional log-likelihood. On Omniglot, as the number of conditioning examples increased from 1 to 19, the GMN's predictive performance improved much more sharply than its competitors.

Qualitative comparison of GMN variants on Omniglot, showing the impact of pseudo-inputs and attention.

Qualitatively, the model produces sharp, recognizable variations of characters it has never seen during training (Figure 6). The authors found that "pseudo-inputs"—trainable parameters acting as a default template—prevent the model from overfitting to the conditioning set. Even though GMNs are generative, they can be used for classification; in a 5-class, 5-shot setting, GMNs achieved 97.4% accuracy, rivaling supervised models.

Limitations and Future Work

While GMNs excel on character strokes, it remains to be seen how well this attentional matching scales to high-resolution, complex natural images. Additionally, "Full Context Matching" involves multiple passes with an RNN, which increases computational overhead. Future work could explore applying GMNs to semi-supervised learning or extending the architecture to handle more complex, hierarchical data structures.

Summary updated

Abstract

Despite recent advances, the remaining bottlenecks in deep generative models are necessity of extensive training and difficulties with generalization from small number of training examples. We deve...
BibTeX
@inproceedings{bartunov2018few,
  author = {Sergey Bartunov and Dmitry P. Vetrov},
  title = {{Few-shot Generative Modelling with Generative Matching Networks}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2018},
  url = {https://openreview.net/forum?id=tzAqFz_LnhM},
}
Figure