Text-to-image Synthesis via Symmetrical Distillation Networks

Yuxin Peng(彭宇新), Mingkuan Yuan, Yuxin Peng

TL;DR

Symmetrical Distillation Networks (SDN) bridge the gap between abstract text descriptions and raw pixels by using a pre-trained image classifier (like VGG19) as a "teacher." By mirroring the teacher's architecture, the generative "student" model learns to produce realistic images through a stable, two-stage distillation process rather than the often-unstable adversarial training used in GANs.

What Problem Does This Paper Address?

Generating a realistic image from a short sentence is a "one-to-many" problem fraught with two major challenges, which the authors define as:

  1. The Heterogeneous Gap: This is the disconnect between high-level semantic concepts (the word "yellow") and low-level pixel values (RGB arrays). Mapping text to pixels is inherently difficult because the data formats are entirely different.
  2. The Homogeneous Gap: This refers to the difference between the distribution of synthetic images and real images. Even if a model gets the "concept" right (e.g., a bird), the resulting image often looks blurry or structurally "off" compared to a real photograph.

While Generative Adversarial Networks (GANs) are the standard tool for this, they are notoriously difficult to train, requiring a delicate balance in a minimax game that often leads to mode collapse or unrealistic local details.

Methods

The authors propose Symmetrical Distillation Networks (SDN). The core idea is to leverage the "knowledge" already stored in a generic discriminative model (a VGG19 trained on ImageNet) to guide a generator.

The Symmetrical Architecture

The system consists of a Source Discriminative Model (Teacher) and a Target Generative Model (Student). The student is built as a mirror image of the teacher: where the teacher has down-sampling convolutional layers to extract features, the student has up-sampling layers to build pixels. This symmetry allows for direct, layer-to-layer knowledge transfer.

Two-Stage Distillation

Instead of a single training pass, the authors use a two-step approach:

  • Stage-I (Global Structure): The student learns to match the teacher’s internal representations of a real image, given only a text embedding. This teaches the student the basic "recipe" for what features (like wings or petals) should look like in the teacher's feature space.
  • Stage-II (Fine Details): The student generates an image, which is then fed into the teacher. The student is penalized based on the difference between the teacher's perception of the synthetic image versus the real image. This forces the student to refine tiny discrepancies in texture and local detail.

Diverse Synthesis

To handle the fact that one sentence can describe many different images, the authors implement "multiple choice learning." The model generates $n$ images simultaneously, but the loss function only updates the "best" one, encouraging the different output channels to specialize in different variations (colors, poses, etc.).

Datasets

The model was evaluated on two gold-standard benchmarks for fine-grained text-to-image synthesis:

  • CUB-200-2011: 11,788 images of 200 bird species with detailed captions.
  • Oxford-Flower-102: 8,189 images of 102 flower categories. The authors used a zero-shot setting, meaning the model was tested on categories it never saw during training, proving its ability to generalize.

Key Insights and Ideas

  • Classifiers as Generative Guides: We usually think of classifiers as "pixel-to-label" machines. This paper flips that, suggesting that the internal layers of a classifier contain a hierarchical roadmap of visual concepts that can be used to "teach" a generator.
  • Stability over Competition: By replacing the adversarial "competition" of a GAN with a "teacher-student" distillation, the training process becomes a feedforward optimization. This avoids the instability and hyperparameter sensitivity common in GANs.
  • Multi-Level Bridging: High-level layers in the teacher bridge the heterogeneous (semantic) gap, while mid- and low-level layers (which capture textures and edges) bridge the homogeneous (authenticity) gap.

Key Results

The SDN model significantly outperformed state-of-the-art models like StackGAN and GAN-INT-CLS across multiple metrics:

  • Inception Score (IS): On the CUB bird dataset, SDN achieved a score of 6.89, compared to StackGAN’s 4.95. A higher score indicates higher quality and diversity.
  • Image Similarity (SSIM/FSIM): SDN consistently scored higher in structural similarity to real images.
  • Human Ranking: In user studies, humans ranked SDN images as having better quality and better correlation with the input text than competing methods.
  • Qualitative Quality: As seen in the paper's figures, SDN produces much sharper local details (like the texture of feathers or the center of a flower) compared to the often-blurry outputs of previous models.

Limitations and Future Work

  • Shape Uncertainty: The authors note that in some failure cases (especially in Stage-I), the model can produce uncertain or "blobby" object shapes because the Stage-I loss is focused more on feature-space alignment than crisp edges.
  • Generic Knowledge Limits: Because the "teacher" (VGG19) is a generic model, it might not capture highly specific or rare attributes mentioned in very complex text descriptions.
  • Future Directions: The authors suggest extending this symmetrical distillation framework to other tasks, such as human pose synthesis or video generation, and moving toward higher-resolution image outputs.

Summary updated

Abstract

Text-to-image synthesis aims to automatically generate images according to text descriptions given by users, which is a highly challenging task. The main issues of text-to-image synthesis lie in two gaps: the heterogeneous and homogeneous gaps. The heterogeneous gap is between the high-level concepts of text descriptions and the pixel-level contents of images, while the homogeneous gap exists between synthetic image distributions and real image distributions. For addressing these problems, we exploit the excellent capability of generic discriminative models (e.g. VGG19), which can guide the training process of a new generative model on multiple levels to bridge the two gaps. The high-level representations can teach the generative model to extract necessary visual information from text descriptions, which can bridge the heterogeneous gap. The mid-level and low-level representations can lead it to learn structures and details of images respectively, which relieves the homogeneous gap. Therefore, we propose Symmetrical Distillation Networks (SDN) composed of a source discriminative model as "teacher" and a target generative model as "student". The target generative model has a symmetrical structure with the source discriminative model, in order to transfer hierarchical knowledge accessibly. Moreover, we decompose the training process into two stages with different distillation paradigms for promoting the performance of the target generative model. Experiments on two widely-used datasets are conducted to verify the effectiveness of our proposed SDN.
BibTeX
@inproceedings{peng2018text,
  author = {Yuxin Peng(彭宇新) and Mingkuan Yuan and Yuxin Peng},
  title = {{Text-to-image Synthesis via Symmetrical Distillation Networks}},
  booktitle = {ACM Multimedia},
  year = {2018},
  doi = {10.1145/3240508.3240559},
}

Loading related content...

Figure