Learning to Memorize Feature Hallucination for One-Shot Image Generation

Yu Xie, Yanwei Fu, Ying Tai, Yun Cao, Junwei Zhu, Chengjie Wang

Abstract

One-shot image generation (OSG) is a challenging task that requires generating diverse, high-quality images of a novel category from a single example. We introduce the Memorize Feature Hallucination (MFH) framework, which addresses this by disentangling category-related features from category-independent attributes (such as pose or background) stored in a dedicated memory module. By recombining these features, MFH "hallucinates" novel variations of unseen objects. Our approach achieves state-of-the-art performance on the NABirds and AnimalFace datasets, significantly improving both image diversity and quality.

Introduction

Generating images typically requires thousands of training examples to capture a category's distribution. One-Shot Image Generation (OSG) represents the extreme limit of this task: given exactly one image of a new category, the model must generate dozens of realistic, diverse variations. Existing methods often suffer from overfitting to the single input or producing low-diversity outputs that look nearly identical to the source.

The MFH framework draws inspiration from human imagination. When humans see a new animal, they can imagine it in different poses because they possess prior knowledge of what "running" or "sitting" looks like from other species. MFH implements this by reusing features across categories, effectively giving AI a library of "imagined" contexts.

One-shot image generation results showing diverse hallucinations of meerkats and birds from a single source image.

Methodology

The core of the MFH model is the explicit separation of "what" an object is from "how" it is presented.

Learning to Memorize (L2M)

The L2M module splits an image's features into two distinct types:

  1. Category-Related (CR): Features defining the object's identity (e.g., the unique markings of a specific bird).
  2. Category-Independent (CI): Class-agnostic features such as pose, background, or specific actions like an open mouth.

These CI features are stored in a Memory Module ($M$), creating a "visual dictionary" of possible appearances built up during a pre-training phase on common categories.

Examples of category-independent features, such as an "open mouth" pose across different species, which can be stored in memory and applied to novel categories.

Feature Hallucination (FeaHa)

To generate a new image, the model extracts the CR feature from the one-shot input and pairs it with a randomly sampled CI feature from its memory. A generator then blends these to produce a "hallucinated" image.

Key technical innovations include a Gumbel-Softmax Addresser for differentiable memory access, Pairwise Supervision to enforce diversity between generated samples, and Split AdaIN to control layout (low-resolution layers) and fine details (high-resolution layers) separately.

Experimental Results

The model was evaluated on the AnimalFace (149 categories) and NABirds (555 categories) datasets. MFH significantly outperformed baselines like MineGAN, FastGAN, and FinetuneGAN across multiple metrics:

  • FID Scores: On the NABirds dataset, MFH achieved an FID of 42.24, a nearly 30% improvement over the next best competitor, FastGAN (59.64).
  • User Preference: In a blind study with 50 participants, MFH was preferred by a wide margin for its superior image quality and diversity.
  • Downstream Utility: Using MFH-generated images for data augmentation improved 5-way 1-shot classification accuracy on NABirds from 77.93% to 82.98%, proving the realism of the hallucinations.

Conclusion and Future Work

MFH demonstrates that disentangling identity from context and utilizing a memory-based prior is a powerful strategy for one-shot generation. While the model occasionally struggles with highly complex poses or occlusions, it sets a new benchmark for "imagining" novel categories. Future research will explore extending this memory-based hallucination to video generation and more complex scene structures.

Summary updated

Abstract

This paper studies the task of One-Shot image Generation (OSG), where generation network learned on base dataset should be generalizable to synthesize images of novel categories with only one available sample per novel category. Most existing methods for feature transfer in oneshot image generation only learn reusable features implicitly on pre-training tasks. Such methods would be likely to overfit pre-training tasks. In this paper, we propose a novel model to explicitly learn and memorize reusable features that can help hallucinate novel category images. To be specific, our algorithm learns to decompose image features into the Category-Related (CR) and Category-Independent(CI) features. Our model learning to memorize class-independent CI features which are further utilized by our feature hallucination component to generate target novel category images. We validate our model on several benchmarks. Extensive experiments demonstrate that our model effectively boosts the OSG performance and can generate compelling and diverse samples.
BibTeX
@inproceedings{xie2022learning,
  author = {Yu Xie and Yanwei Fu and Ying Tai and Yun Cao and Junwei Zhu and Chengjie Wang},
  title = {{Learning to Memorize Feature Hallucination for One-Shot Image Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00892},
}
Figure