LoFGAN: Fusing Local Representations for Few-shot Image Generation

Zheng Gu, Wenbin Li, Jing Huo, Lei Wang, Yang Gao

Introduction

Deep generative models, particularly Generative Adversarial Networks (GANs), have demonstrated remarkable success in synthesizing high-quality images. However, their performance typically relies on the availability of massive datasets. In "few-shot" scenarios—where only a handful of examples (e.g., 3 or 5) are available for a specific category—standard GANs often fail to capture the underlying distribution, leading to poor quality and limited diversity.

Previous attempts to address few-shot generation often utilized "fusion-based" approaches, which blend the features of available images globally. A significant drawback of global fusion is semantic misalignment. For instance, if input images feature faces in different orientations, a global average results in blurry "ghosting" artifacts. Furthermore, global fusion restricts the model to generating simple linear combinations of inputs, stifling diversity. LoFGAN (Local Fusion GAN) overcomes these hurdles by shifting the focus from global blending to local representation matching and fusion.

Methodology: The Local Fusion Module (LFM)

The core contribution of LoFGAN is the Local Fusion Module (LFM). Instead of a global weighted average, the LFM employs a "match-and-replace" strategy to synthesize new feature maps from a set of input images.

The LoFGAN architecture (left) and the detailed Local Fusion Module (right). The pipeline includes an encoder $E$, the LFM, a decoder $H$, and a discriminator $D$.

The LFM process follows these key steps:

  1. Base and Reference Partitioning: Given $k$ input images, one is randomly designated as the "base" image to provide the structural layout, while the remaining images serve as "references" providing a pool of local features.
  2. Local Selection: The model identifies specific spatial positions within the base image's feature map to be modified.
  3. Semantic Matching: For each selected position, the module searches the reference images for the most semantically similar local features using cosine similarity. This ensures that a "petal" feature is replaced by another "petal" feature, maintaining semantic integrity.
  4. Local Replacement: The original base features at the selected positions are replaced with a fusion of the best-matched reference features.
  5. Local Reconstruction Loss: To stabilize training and provide sharp supervision, the authors introduce a novel loss function ($\\mathcal{L}_{local}$). This loss encourages the GAN to reconstruct a target image formed by performing the same local swapping at the pixel level, rather than a blurry global average.

Key Insights: Local vs. Global Fusion

The fundamental insight of LoFGAN is that while two images may be globally dissimilar, their local components (e.g., textures, specific parts) are often highly compatible. By exploiting this local compatibility, LoFGAN can generate "hybrid" images that are both realistic and distinct from any single input image.

Visualization of the Local Fusion Module logic. Unlike a weighted stack (right) which causes blurriness, LFM (center) replaces local patches to maintain sharpness and semantic structure.

A critical hyperparameter in this process is the selection rate ($\\eta$), which determines the proportion of the image to be swapped. The researchers found a "Goldilocks zone" around $\\eta = 0.5$; swapping too little results in outputs too similar to the base image, while swapping too much can lead to structural instability.

Experimental Evaluation

LoFGAN was evaluated on three challenging datasets: Flowers (high intra-class variance), Animal Faces (complex textures), and VGGFace (highly structured geometry).

Quantitative Results

LoFGAN achieved state-of-the-art performance across multiple metrics:

  • Image Quality (FID): On the Flowers dataset, LoFGAN reached an FID of 79.33, significantly outperforming the previous state-of-the-art, F2GAN, which scored 120.48.
  • Diversity (LPIPS): The model consistently produced more diverse images than baseline methods, demonstrating its ability to explore the generation space more effectively through local permutations.
  • Downstream Utility: Using LoFGAN-generated images for data augmentation improved top-1 classification accuracy on the Flowers dataset from 60.0% to 65.1%, proving the functional realism of the synthetic data.

Conclusion

LoFGAN represents a significant step forward in few-shot image generation by prioritizing local semantic alignment over global feature blending. By matching and fusing compatible local representations, it produces high-quality, diverse images that are useful for downstream tasks. While challenges remain—such as dependency on the base image's layout and the difficulty of 1-shot generation—the local fusion paradigm offers a robust framework for learning from limited data.

Summary updated

Abstract

Given only a few available images for a novel unseen category, few-shot image generation aims to generate more data for this category. Previous works attempt to globally fuse these images by using adjustable weighted coefficients. However, there is a serious semantic misalignment between different images from a global perspective, making these works suffer from poor generation quality and diversity. To tackle this problem, we propose a novel Local-Fusion Generative Adversarial Network (LoFGAN) for fewshot image generation. Instead of using these available images as a whole, we first randomly divide them into a base image and several reference images. Next, LoFGAN matches local representations between the base and reference images based on semantic similarities, and replaces the local features with the closest related local features. In this way, LoFGAN can produce more realistic and diverse images at a more fine-grained level, and simultaneously enjoy the characteristic of semantic alignment. Furthermore, a local reconstruction loss is also proposed, which can provide better training stability and generation quality. We conduct extensive experiments on three datasets, which successfully demonstrates the effectiveness of our proposed method for few-shot image generation and downstream visual applications with limited data. Code is available at https://github.com/edward3862/LoFGAN-pytorch.
BibTeX
@inproceedings{gu2021lofgan,
  author = {Zheng Gu and Wenbin Li and Jing Huo and Lei Wang and Yang Gao},
  title = {{LoFGAN: Fusing Local Representations for Few-shot Image Generation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00835},
}
Figure