Abstract
Semantic image synthesis—generating photorealistic images from pixel-level label maps—traditionally requires massive datasets with meticulously annotated masks. This paper introduces Class Affinity Transfer (CAT), a novel framework designed to enable high-quality synthesis in low-data regimes (as few as 100 training images). By prepending a "class affinity matrix" to pre-trained GAN or Diffusion models, CAT maps knowledge from large-scale source datasets to target domains with different label spaces. Our results demonstrate that CAT significantly improves both image quality (FID) and semantic accuracy (mIoU) compared to standard transfer learning baselines, effectively reducing the human labor required for pixel-precise annotations.
Introduction
State-of-the-art semantic image synthesis models, such as OASIS and SPADE, have achieved remarkable results but remain notoriously data-hungry. The primary bottleneck is the annotation cost: creating a single pixel-precise semantic map for a dataset like Cityscapes can require over 1.5 hours of human labor. While transfer learning is a standard solution in image classification, it is significantly more complex in conditional generation. The "label space"—the specific list of object categories—often changes between datasets. A model trained on "forests" and "rivers" does not inherently know how to interpret a label for "parks" or "ponds," often forcing researchers to retrain models from scratch and discard valuable prior knowledge about visual textures and structures.
Methodology: Class Affinity Transfer (CAT)
The authors propose CAT as a bridge between a source model (trained on a large dataset) and a target domain (a small, few-shot dataset). The method focuses on aligning the disparate label spaces to leverage pre-existing generative capabilities.
1. The Class Affinity Matrix ($A$)
The core innovation is a linear layer, the Class Affinity Matrix $A$, prepended to the source model. This matrix maps target classes to source classes. For instance, if the target class is "sunflower," the matrix might instruct the model to treat it as a weighted combination of "flower" and "plant" from its original training. This allows the model to reuse its learned weights for similar visual concepts.
2. Estimating Affinities
To avoid manual mapping, the researchers developed three automated ways to populate the affinity matrix:
- Supervised Segmentation: Utilizing a pre-trained segmenter to observe how it labels target images using source categories.
- Self-supervised Features: Using visual "prototypes" (via iBOT) to identify classes that share visual similarities.
- Text-based Embeddings: Leveraging CLIP to find classes that are semantically or linguistically related (e.g., "car" and "truck").
A majority voting scheme that combines these three modalities was found to yield the most robust mappings.
3. Architectural Integration
CAT was integrated into two distinct generative backbones:
- Generative Adversarial Networks (OASIS): The researchers modified the SPADE blocks to include the affinity matrix and introduced residual branches. These branches allow the model to learn entirely new features that may not have existed in the source dataset.
- Diffusion Models (PITI): The class-to-RGB mapping was factored into the first layer of the encoder, and trainable "prompts" were introduced into the transformer blocks to facilitate better domain adaptation.
Experimental Results
The study utilized major benchmarks including ADE20K (20k images) and COCO-Stuff (110k images) as source datasets, with small subsets (25, 100, or 400 images) of ADE20K, COCO-Stuff, and Cityscapes as targets.
Quantitative Performance
On the COCO → ADE transfer task using only 100 images, CAT demonstrated substantial improvements:
- Image Quality: The Fréchet Inception Distance (FID) improved from 54.0 (random initialization) to 40.9.
- Semantic Accuracy: For the PITI diffusion model, the mean Intersection over Union (mIoU) jumped from 11.6 to 22.3, indicating that the generated images adhered much more closely to the input label maps.
The "Training-Free" Phenomenon
A striking finding was that CAT works reasonably well even before fine-tuning. By simply plugging in the affinity matrix, the model can generate recognizable images for the new dataset's labels. This suggests that the "knowledge" of how to render objects is largely transferable if the labels can be aligned correctly.
Discussion and Limitations
CAT provides a "warm start" for models in low-data regimes, preventing the catastrophic forgetting often seen when fine-tuning on very small datasets. However, some challenges remain:
- Semantic Aberrations: Automated mapping can occasionally fail. For example, a "truck" in Cityscapes might be mapped to a "building" in ADE20K because both often appear as large, rectangular, gray blocks in their respective datasets.
- Domain Gaps: While CAT handles class differences well, it is less effective at bridging massive stylistic gaps, such as moving from the clear, sunny conditions of COCO to the foggy, overcast environment of Cityscapes.
Conclusion
Class Affinity Transfer represents a significant step toward making semantic image synthesis more accessible by drastically reducing the need for large-scale annotated datasets. By treating label alignment as a cross-modal mapping problem, CAT allows existing high-capacity models to be repurposed for niche or emerging domains with minimal human intervention. Future work will likely focus on better handling "out-of-vocabulary" classes that have no visual or semantic counterparts in the source data.
Summary updated