Abstract
One-shot image generation (OSG) is a challenging task that requires generating diverse, high-quality images of a novel category from a single example. We introduce the Memorize Feature Hallucination (MFH) framework, which addresses this by disentangling category-related features from category-independent attributes (such as pose or background) stored in a dedicated memory module. By recombining these features, MFH "hallucinates" novel variations of unseen objects. Our approach achieves state-of-the-art performance on the NABirds and AnimalFace datasets, significantly improving both image diversity and quality.
Introduction
Generating images typically requires thousands of training examples to capture a category's distribution. One-Shot Image Generation (OSG) represents the extreme limit of this task: given exactly one image of a new category, the model must generate dozens of realistic, diverse variations. Existing methods often suffer from overfitting to the single input or producing low-diversity outputs that look nearly identical to the source.
The MFH framework draws inspiration from human imagination. When humans see a new animal, they can imagine it in different poses because they possess prior knowledge of what "running" or "sitting" looks like from other species. MFH implements this by reusing features across categories, effectively giving AI a library of "imagined" contexts.
Methodology
The core of the MFH model is the explicit separation of "what" an object is from "how" it is presented.
Learning to Memorize (L2M)
The L2M module splits an image's features into two distinct types:
- Category-Related (CR): Features defining the object's identity (e.g., the unique markings of a specific bird).
- Category-Independent (CI): Class-agnostic features such as pose, background, or specific actions like an open mouth.
These CI features are stored in a Memory Module ($M$), creating a "visual dictionary" of possible appearances built up during a pre-training phase on common categories.
Feature Hallucination (FeaHa)
To generate a new image, the model extracts the CR feature from the one-shot input and pairs it with a randomly sampled CI feature from its memory. A generator then blends these to produce a "hallucinated" image.
Key technical innovations include a Gumbel-Softmax Addresser for differentiable memory access, Pairwise Supervision to enforce diversity between generated samples, and Split AdaIN to control layout (low-resolution layers) and fine details (high-resolution layers) separately.
Experimental Results
The model was evaluated on the AnimalFace (149 categories) and NABirds (555 categories) datasets. MFH significantly outperformed baselines like MineGAN, FastGAN, and FinetuneGAN across multiple metrics:
- FID Scores: On the NABirds dataset, MFH achieved an FID of 42.24, a nearly 30% improvement over the next best competitor, FastGAN (59.64).
- User Preference: In a blind study with 50 participants, MFH was preferred by a wide margin for its superior image quality and diversity.
- Downstream Utility: Using MFH-generated images for data augmentation improved 5-way 1-shot classification accuracy on NABirds from 77.93% to 82.98%, proving the realism of the hallucinations.
Conclusion and Future Work
MFH demonstrates that disentangling identity from context and utilizing a memory-based prior is a powerful strategy for one-shot generation. While the model occasionally struggles with highly complex poses or occlusions, it sets a new benchmark for "imagining" novel categories. Future research will explore extending this memory-based hallucination to video generation and more complex scene structures.
Summary updated