Matchinggan: Matching-Based Few-Shot Image Generation

Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang

TL;DR

This paper introduces MatchingGAN, a novel approach to few-shot image generation that leverages a "matching procedure" to effectively fuse information from a small set of conditional images. By learning to match random vectors with these conditional images, MatchingGAN can generate diverse and realistic new images for categories with very limited training data, outperforming existing methods in both generation quality and downstream classification tasks.

What Problem Does This Paper Address?

Generating realistic images typically requires vast amounts of training data. However, in many real-world scenarios, obtaining such large datasets for specific categories is prohibitively expensive or simply impossible. This is the challenge of "few-shot generation," where the goal is to create new, high-quality images from only a handful of examples. Existing methods often struggle to produce diverse and realistic images, or they require extensive fine-tuning on unseen categories. This paper aims to bridge this gap by developing a method that can effectively learn from and generate images for categories with minimal data.

Methods

The core innovation of MatchingGAN lies in its "matching procedure" integrated into a Generative Adversarial Network (GAN) framework. The model consists of two main components:

  • Matching Generator: This component takes a random vector and a few conditional images as input. It uses encoders to project both the random vector and the conditional images into a common "matching space." Here, similarity scores are computed between the random vector and each conditional image. These scores act as interpolation coefficients, allowing the generator to intelligently fuse features from the conditional images. This fusion process is designed to capture the variance within the limited data. A weighted reconstruction loss is also employed to ensure the generated image is similar to the conditional images based on their similarity scores.
  • Matching Discriminator: This discriminator not only distinguishes real from fake images (standard GAN practice) but also incorporates a "feature matching" component. It ensures that the generated image's discriminative features align with the fused discriminative features of the conditional images. This helps to maintain the interpolated information from the conditional images within the generated output. The discriminator also includes a classifier to distinguish between different categories.

The overall framework combines adversarial learning with this novel matching procedure to achieve robust few-shot generation.

Datasets

The authors evaluated their MatchingGAN on three standard datasets:

  • Omniglot: A dataset of handwritten characters from various alphabets.
  • EMNIST: A dataset of handwritten digits and letters.
  • VGGFace: A dataset of human faces.

They followed existing benchmarks for splitting categories into "seen" (for training) and "unseen" (for testing) sets, and for cases with more than 100 samples per category, they randomly selected 100 to simulate a low-data setting.

Key Insights and Ideas

  • The Power of "Matching": The paper convincingly argues that learning to "match" a random latent vector to conditional images is crucial for effective few-shot generation. This matching process allows the model to learn how to weigh information from different conditional images, leading to more coherent and realistic outputs.
  • Bridging Generation and Classification: By incorporating a classification loss into the discriminator and using generated images for downstream classification tasks, the authors demonstrate that their generation method directly benefits classification performance, highlighting the utility of their generated data.
  • Feature Fusion is Key: The success of MatchingGAN hinges on its ability to fuse multi-level information from conditional images. The use of skip connections in the generator's UNet-like architecture plays a significant role in this coherent fusion.
  • Beyond Single-Image Conditioning: Unlike some prior works that rely on a single conditional image, MatchingGAN's ability to leverage multiple conditional images is a key differentiator, enabling richer and more diverse generation.

Key Results

  • Generation Quality: MatchingGAN achieved the highest Inception Score (IS) and lowest Fréchet Inception Distance (FID) on the VGGFace dataset compared to baseline methods like FIGR, GMN, and DAGAN, indicating superior generation quality and diversity.
  • Classification Performance: When used for data augmentation in low-data and few-shot classification settings, MatchingGAN consistently outperformed baseline methods, demonstrating the practical utility of its generated images. For instance, on the EMNIST dataset in a low-data setting, MatchingGAN achieved an accuracy of 91.75% with 5 conditional images, significantly higher than other methods.
  • Ablation Studies: Experiments showed that the matching procedure is essential, outperforming random interpolation. The number of skip connections in the generator also impacts performance, with two connections offering a good balance between realism and diversity.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores hyperparameter tuning (e.g., $\lambda_r$, $\lambda_m$), it acknowledges that larger $\lambda_r$ can lead to lower FID and higher IS at the cost of classification performance, suggesting a trade-off.
  • Generalization of Matching: The paper conjectures that if MatchingGAN is trained with $K_1$ conditional images, it might not generalize as well to fusing information from more conditional images ($K_2 > K_1$) during testing. This suggests potential limitations in adapting to varying numbers of conditional images at inference time.
  • Future Work: The authors suggest exploring different network designs and further investigating the impact of hyperparameter choices. The paper also implicitly points towards future work in understanding the theoretical underpinnings of the matching procedure and its generalization capabilities.

Summary updated

Abstract

To generate new images for a given category, most deep generative models require abundant training images from this category, which are often too expensive to acquire. To achieve the goal of generation based on only a few images, we propose matching-based Generative Adversarial Network (GAN) for few-shot generation, which includes a matching generator and a matching discriminator. Matching generator can match random vectors with a few conditional images from the same category and generate new images for this category based on the fused features. The matching discriminator extends conventional GAN discriminator by matching the feature of generated image with the fused feature of conditional images. Extensive experiments on three datasets demonstrate the effectiveness of our proposed method.
BibTeX
@inproceedings{hong2020matchinggan,
  author = {Yan Hong and Li Niu and Jianfu Zhang and Liqing Zhang},
  title = {{Matchinggan: Matching-Based Few-Shot Image Generation}},
  booktitle = {ICME},
  year = {2020},
  doi = {10.1109/icme46284.2020.9102917},
}
Figure