Augmentation-Interpolative AutoEncoders for Unsupervised Few-Shot Image Generation

Davis Wertheimer, Omid Poursaeed, Bharath Hariharan

Abstract

We aim to build image generation models that generalize to new domains from few examples. To this end, we first investigate the generalization properties of classic image generators, and discover that autoencoders generalize extremely well to new domains, even when trained on highly constrained data. We leverage this insight to produce a robust, unsupervised few-shot image generation algorithm, and introduce a novel training procedure based on recovering an image from data augmentations. Our Augmentation-Interpolative AutoEncoders synthesize realistic images of novel objects from only a few reference images, and outperform both prior interpolative models and supervised few-shot image generators. Our procedure is simple and lightweight, generalizes broadly, and requires no category labels or other supervision during training.

Introduction

Modern generative models, such as GANs and VAEs, are notoriously data-hungry. Synthesizing high-quality images typically requires thousands of labeled examples per category. This requirement creates a significant bottleneck for "few-shot" scenarios, such as generating concept art for a unique object or creating synthetic data for rare medical conditions.

A fundamental issue identified in this work is that powerful models like VAEs and GANs are designed to match a specific training distribution (the "prior"). While this makes them excellent at memorizing training classes, it severely limits their ability to represent novel concepts. When presented with a "novel" image—for instance, a baby's face when trained only on adult faces—these models often "hallucinate" training features, failing to capture the unique characteristics of the new domain.

Comparison of face generation on training vs. novel data. While PGAN (red box) fails to generalize to novel classes (babies), AugIntAE (green box) maintains high fidelity.

Methodology: Augmentation-Interpolative AutoEncoders

The researchers observed that simple AutoEncoders (AEs) generalize to new domains far better than VAEs or GANs because they do not enforce a rigid latent distribution. To transform an AE into a generator, the authors developed Augmentation-Interpolative AutoEncoders (AugIntAE).

AugIntAE builds upon Interpolative AutoEncoders (IntAEs), which are trained to create images by interpolating between latent codes. Standard IntAEs often fail on novel data because they attempt to force every interpolation to resemble the training set. AugIntAE solves this by using data augmentation to create a "ground truth" for interpolation:

  1. Augmentation: A single image $x$ is subjected to two different sets of augmentations (e.g., rotation, color jitter) to produce images $A$ and $B$.
  2. Target Creation: A third image $C$ is created using a "middle-ground" version of these augmentations.
  3. Training: The model is trained to reconstruct image $C$ by interpolating the latent codes of $A$ and $B$.

By training on known transformations, the model learns the underlying logic of image changes (e.g., how a shape rotates) rather than just memorizing class-specific features. This is supplemented by a standard reconstruction loss and a lightweight adversarial loss to enhance image sharpness.

Experimental Evaluation

The model was evaluated across three distinct regimes to test its generalization capabilities:

  • Handwritten Characters: Training on MNIST (digits) and testing on EMNIST (letters); training on Omniglot alphabets and testing on unseen ones.
  • Faces: Training on male faces from Celeb-A and generating female faces or infants.
  • General Objects: Training on CIFAR-10 and evaluating on the more diverse CIFAR-100.

Key Results and Insights

AugIntAE consistently outperformed both standard interpolative models and complex supervised few-shot generators like DAGAN.

  • Quantitative Performance: On the MNIST $\rightarrow$ EMNIST task, AugIntAE achieved an FID score of 12.3, significantly outperforming standard IntAE (21.5) and WGAN-GP (25.0).
  • Qualitative Fidelity: The model produced smooth interpolations without the "ghosting" artifacts common in other models. It successfully preserved the semantic shape of novel characters and faces.
  • Unsupervised Learning: A major advantage of AugIntAE is that it requires no category labels during training, making it highly versatile for uncurated datasets.
Generalization from digits (MNIST) to letters (EMNIST). AugIntAE reconstructions (encoder) closely match the source characters, even for unseen classes.

The core insight is that the "prior" in traditional generative models acts as a constraint that prevents generalization. By relying on the inherent flexibility of autoencoders and learning semantic trajectories through augmentation, AugIntAE effectively "navigates" the latent space of novel classes.

Limitations and Future Work

While AugIntAE represents a significant step forward, some challenges remain. The model can oversimplify highly complex or irregular shapes. Additionally, outputs for natural images (like CIFAR-100) can sometimes be blurry. Future work will focus on improving fidelity for complex scenes and developing more sophisticated sampling methods that do not rely on existing "seed" images for interpolation.

Summary updated

Abstract

We aim to build image generation models that generalize to new domains from few examples. To this end, we first investigate the generalization properties of classic image generators, and discover that autoencoders generalize extremely well to new domains, even when trained on highly constrained data. We leverage this insight to produce a robust, unsupervised few-shot image generation algorithm, and introduce a novel training procedure based on recovering an image from data augmentations. Our Augmentation-Interpolative AutoEncoders synthesize realistic images of novel objects from only a few reference images, and outperform both prior interpolative models and supervised few-shot image generators. Our procedure is simple and lightweight, generalizes broadly, and requires no category labels or other supervision during training.
BibTeX
@misc{wertheimer2020augmentation,
  author = {Davis Wertheimer and Omid Poursaeed and Bharath Hariharan},
  title = {{Augmentation-Interpolative AutoEncoders for Unsupervised Few-Shot Image Generation}},
  year = {2020},
  eprint = {2011.13026},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.13026},
}
Figure