Adversarial Learning of Semantic Relevance in Text to Image Synthesis

Miriam Cha, Youngjune L. Gwon, H. T. KungAAAI 2019 · DOI

Abstract

Generative Adversarial Networks (GANs) for text-to-image synthesis often struggle with mode collapse and the inherent ambiguity of one-to-many mappings. We present Text-SeGAN, a novel framework that replaces traditional discrete class-label predictors with a semantic relevance regressor. By utilizing a curriculum-based "easy-to-hard" negative sampling strategy, Text-SeGAN encourages the generator to explore a continuous relevance space rather than being confined to rigid categories. Experimental results on the Oxford-102 flower dataset demonstrate that Text-SeGAN significantly improves both image quality and diversity, achieving a state-of-the-art Inception Score of 4.03.

Introduction

The synthesis of realistic images from natural language descriptions is a challenging task in computer vision. While GANs have shown promise, two major issues persist: mode collapse and the one-to-many ambiguity. Mode collapse occurs when the generator produces a limited set of repetitive outputs that successfully fool the discriminator. The one-to-many problem arises because a single text prompt, such as "a yellow flower," can accurately describe numerous distinct species and orientations. Previous approaches, such as TAC-GAN, attempted to mitigate this by forcing the discriminator to classify generated images into specific categories. However, this rigid classification often acts as a bottleneck, restricting the generator's diversity and leading back to mode collapse.

Methodology

Text-SeGAN addresses these limitations through two key innovations: Semantic Relevance Regression and Curriculum-based Negative Sampling.

Semantic Relevance Regression

Unlike traditional Conditional GANs (CGANs) or Twin Auxiliary Classifier GANs (TAC-GANs) that use discrete class labels, Text-SeGAN's discriminator incorporates a regressor. This regressor outputs a fractional value between 0 and 1, representing the semantic alignment between the image and the text description. This allows the model to learn a continuous "relevance" manifold, facilitating a more natural one-to-many mapping.

Comparison of various GAN architectures for text-to-image synthesis, highlighting the shift to semantic relevance in the proposed method.

Easy-to-Hard Negative Sampling

To refine the discriminator's understanding of semantic relevance, the authors employ a curriculum learning strategy for negative sampling. Training begins with "easy negatives"—images whose descriptions are semantically distant from the target text. As training progresses, "hard negatives"—images that are semantically similar but technically incorrect—are introduced. This gradual increase in difficulty prevents early collapse and forces the model to capture fine-grained semantic details.

Experimental Results

The model was evaluated on the Oxford-102 flower dataset, consisting of 8,189 images across 102 species. Text-SeGAN achieved an Inception Score (IS) of 4.03, a substantial improvement over GAN-INT-CLS (2.66) and TAC-GAN (3.45).

Qualitative comparison of image generation diversity. G1 and G2 represent baseline models showing mode collapse, while G3 (Text-SeGAN) produces diverse outputs for the same prompt.

As shown in the qualitative analysis, while baseline models often produce nearly identical images for a given prompt, Text-SeGAN generates flowers with varied shapes, petal counts, and orientations while maintaining strict adherence to the text's semantic features. Ablation studies confirmed that the curriculum sampling strategy alone boosted the Inception Score from 3.65 to 4.03.

Conclusion

Text-SeGAN demonstrates that moving away from rigid class labels toward a continuous semantic relevance space significantly enhances the performance of text-to-image GANs. The combination of relevance regression and curriculum-based negative sampling effectively mitigates mode collapse. Future research will explore scaling this framework to higher resolutions and more complex, multi-object datasets.

Summary updated

BibTeX
@inproceedings{cha2019adversarial,
  author = {Miriam Cha and Youngjune L. Gwon and H. T. Kung},
  title = {{Adversarial Learning of Semantic Relevance in Text to Image Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33013272},
}
Figure