Abstract
Text-to-image synthesis aims to automatically generate images according to text descriptions given by users, which is a highly challenging task. The main issues of text-to-image synthesis lie in two gaps: the heterogeneous and homogeneous gaps. The heterogeneous gap is between the high-level concepts of text descriptions and the pixel-level contents of images, while the homogeneous gap exists between synthetic image distributions and real image distributions. For addressing these problems, we exploit the excellent capability of generic discriminative models (e.g. VGG19), which can guide the training process of a new generative model on multiple levels to bridge the two gaps. The high-level representations can teach the generative model to extract necessary visual information from text descriptions, which can bridge the heterogeneous gap. The mid-level and low-level representations can lead it to learn structures and details of images respectively, which relieves the homogeneous gap. Therefore, we propose Symmetrical Distillation Networks (SDN) composed of a source discriminative model as "teacher" and a target generative model as "student". The target generative model has a symmetrical structure with the source discriminative model, in order to transfer hierarchical knowledge accessibly. Moreover, we decompose the training process into two stages with different distillation paradigms for promoting the performance of the target generative model. Experiments on two widely-used datasets are conducted to verify the effectiveness of our proposed SDN.
Introduction
Generating realistic images from natural language descriptions is a "one-to-many" problem that requires bridging two fundamental gaps. First, the Heterogeneous Gap represents the disconnect between high-level semantic concepts (e.g., the word "yellow") and low-level pixel values (RGB arrays). Second, the Homogeneous Gap refers to the distributional difference between synthetic and real images, often manifesting as blurriness or structural inaccuracies in synthetic outputs. While Generative Adversarial Networks (GANs) are the standard tool for this, they are notoriously difficult to train, often requiring a delicate minimax balance that can lead to mode collapse.
Methodology: Symmetrical Distillation Networks
The proposed Symmetrical Distillation Networks (SDN) framework replaces adversarial competition with a teacher-student distillation paradigm. By leveraging a pre-trained discriminative model (Source Discriminative Model, such as VGG19), the system provides a hierarchical roadmap for the generative model (Target Generative Model).
Symmetrical Architecture
To facilitate efficient knowledge transfer, the student model is designed as a mirror image of the teacher. As the teacher extracts features through down-sampling convolutional layers, the student synthesizes pixels through corresponding up-sampling layers. This symmetry allows for direct, layer-to-layer knowledge transfer.
Two-Stage Distillation
The training is divided into two distinct phases to promote stability and quality:
- Stage-I (Global Structure): The student learns to map text embeddings to the teacher's internal feature space using real images as a reference. This teaches the student the basic visual "recipe" for what features (like wings or petals) should look like.
- Stage-II (Fine Details): The student generates synthetic images, which are then processed by the teacher. The student is optimized to minimize the difference between the teacher's perception of its synthetic output and the real target image, forcing the model to refine textures and local details.
To handle the inherent diversity in text descriptions, the authors also implement "multiple choice learning," where the model generates several variations but only updates based on the best-performing output.
Experiments and Results
The SDN model was evaluated on the CUB-200-2011 (birds) and Oxford-Flower-102 datasets. Quantitative results show that SDN significantly outperforms previous state-of-the-art methods. On the CUB dataset, SDN achieved an Inception Score (IS) of 6.89, a substantial improvement over StackGAN's 4.95. Higher IS indicates both better image quality and greater diversity.
Qualitatively, SDN produces images with sharper local details, such as feather textures and distinct flower centers, compared to the often-blurry outputs of GAN-based baselines. Human user studies further confirmed that SDN-generated images correlate better with input text descriptions and possess higher visual authenticity.
Conclusion
SDN demonstrates that pre-trained discriminative models can serve as powerful guides for generative tasks. By bridging the heterogeneous and homogeneous gaps through multi-level distillation, SDN provides a stable and effective alternative to traditional adversarial training for text-to-image synthesis. Future work may extend this framework to higher-resolution outputs and more complex synthesis tasks like video generation.
Summary updated