Text-to-image Synthesis via Symmetrical Distillation Networks

Mingkuan Yuan, Yuxin Peng

Abstract

Text-to-image synthesis aims to automatically generate images according to text descriptions given by users, which is a highly challenging task. The main issues of text-to-image synthesis lie in two gaps: the heterogeneous and homogeneous gaps. The heterogeneous gap is between the high-level concepts of text descriptions and the pixel-level contents of images, while the homogeneous gap exists between synthetic image distributions and real image distributions. For addressing these problems, we exploit the excellent capability of generic discriminative models (e.g. VGG19), which can guide the training process of a new generative model on multiple levels to bridge the two gaps. The high-level representations can teach the generative model to extract necessary visual information from text descriptions, which can bridge the heterogeneous gap. The mid-level and low-level representations can lead it to learn structures and details of images respectively, which relieves the homogeneous gap. Therefore, we propose Symmetrical Distillation Networks (SDN) composed of a source discriminative model as "teacher" and a target generative model as "student". The target generative model has a symmetrical structure with the source discriminative model, in order to transfer hierarchical knowledge accessibly. Moreover, we decompose the training process into two stages with different distillation paradigms for promoting the performance of the target generative model. Experiments on two widely-used datasets are conducted to verify the effectiveness of our proposed SDN.

Introduction

Generating realistic images from natural language descriptions is a "one-to-many" problem that requires bridging two fundamental gaps. First, the Heterogeneous Gap represents the disconnect between high-level semantic concepts (e.g., the word "yellow") and low-level pixel values (RGB arrays). Second, the Homogeneous Gap refers to the distributional difference between synthetic and real images, often manifesting as blurriness or structural inaccuracies in synthetic outputs. While Generative Adversarial Networks (GANs) are the standard tool for this, they are notoriously difficult to train, often requiring a delicate minimax balance that can lead to mode collapse.

Methodology: Symmetrical Distillation Networks

The proposed Symmetrical Distillation Networks (SDN) framework replaces adversarial competition with a teacher-student distillation paradigm. By leveraging a pre-trained discriminative model (Source Discriminative Model, such as VGG19), the system provides a hierarchical roadmap for the generative model (Target Generative Model).

Symmetrical Architecture

To facilitate efficient knowledge transfer, the student model is designed as a mirror image of the teacher. As the teacher extracts features through down-sampling convolutional layers, the student synthesizes pixels through corresponding up-sampling layers. This symmetry allows for direct, layer-to-layer knowledge transfer.

The Symmetrical Distillation Network (SDN) architecture, showing the teacher-student relationship and the two-stage distillation process.

Two-Stage Distillation

The training is divided into two distinct phases to promote stability and quality:

  • Stage-I (Global Structure): The student learns to map text embeddings to the teacher's internal feature space using real images as a reference. This teaches the student the basic visual "recipe" for what features (like wings or petals) should look like.
  • Stage-II (Fine Details): The student generates synthetic images, which are then processed by the teacher. The student is optimized to minimize the difference between the teacher's perception of its synthetic output and the real target image, forcing the model to refine textures and local details.

To handle the inherent diversity in text descriptions, the authors also implement "multiple choice learning," where the model generates several variations but only updates based on the best-performing output.

Experiments and Results

The SDN model was evaluated on the CUB-200-2011 (birds) and Oxford-Flower-102 datasets. Quantitative results show that SDN significantly outperforms previous state-of-the-art methods. On the CUB dataset, SDN achieved an Inception Score (IS) of 6.89, a substantial improvement over StackGAN's 4.95. Higher IS indicates both better image quality and greater diversity.

Qualitative comparison of SDN against state-of-the-art models (StackGAN, GAWWN, GAN-INT-CLS) on the CUB bird dataset.

Qualitatively, SDN produces images with sharper local details, such as feather textures and distinct flower centers, compared to the often-blurry outputs of GAN-based baselines. Human user studies further confirmed that SDN-generated images correlate better with input text descriptions and possess higher visual authenticity.

Conclusion

SDN demonstrates that pre-trained discriminative models can serve as powerful guides for generative tasks. By bridging the heterogeneous and homogeneous gaps through multi-level distillation, SDN provides a stable and effective alternative to traditional adversarial training for text-to-image synthesis. Future work may extend this framework to higher-resolution outputs and more complex synthesis tasks like video generation.

Summary updated

Abstract

Text-to-image synthesis aims to automatically generate images according to text descriptions given by users, which is a highly challenging task. The main issues of text-to-image synthesis lie in two gaps: the heterogeneous and homogeneous gaps. The heterogeneous gap is between the high-level concepts of text descriptions and the pixel-level contents of images, while the homogeneous gap exists between synthetic image distributions and real image distributions. For addressing these problems, we exploit the excellent capability of generic discriminative models (e.g. VGG19), which can guide the training process of a new generative model on multiple levels to bridge the two gaps. The high-level representations can teach the generative model to extract necessary visual information from text descriptions, which can bridge the heterogeneous gap. The mid-level and low-level representations can lead it to learn structures and details of images respectively, which relieves the homogeneous gap. Therefore, we propose Symmetrical Distillation Networks (SDN) composed of a source discriminative model as "teacher" and a target generative model as "student". The target generative model has a symmetrical structure with the source discriminative model, in order to transfer hierarchical knowledge accessibly. Moreover, we decompose the training process into two stages with different distillation paradigms for promoting the performance of the target generative model. Experiments on two widely-used datasets are conducted to verify the effectiveness of our proposed SDN.
BibTeX
@inproceedings{yuan2018text,
  author = {Mingkuan Yuan and Yuxin Peng},
  title = {{Text-to-image Synthesis via Symmetrical Distillation Networks}},
  booktitle = {ACM Multimedia},
  year = {2018},
  doi = {10.1145/3240508.3240559},
}
Figure