Abstract
Generative Adversarial Networks (GANs) for text-to-image synthesis often struggle with mode collapse and the inherent ambiguity of one-to-many mappings. We present Text-SeGAN, a novel framework that replaces traditional discrete class-label predictors with a semantic relevance regressor. By utilizing a curriculum-based "easy-to-hard" negative sampling strategy, Text-SeGAN encourages the generator to explore a continuous relevance space rather than being confined to rigid categories. Experimental results on the Oxford-102 flower dataset demonstrate that Text-SeGAN significantly improves both image quality and diversity, achieving a state-of-the-art Inception Score of 4.03.
Introduction
The synthesis of realistic images from natural language descriptions is a challenging task in computer vision. While GANs have shown promise, two major issues persist: mode collapse and the one-to-many ambiguity. Mode collapse occurs when the generator produces a limited set of repetitive outputs that successfully fool the discriminator. The one-to-many problem arises because a single text prompt, such as "a yellow flower," can accurately describe numerous distinct species and orientations. Previous approaches, such as TAC-GAN, attempted to mitigate this by forcing the discriminator to classify generated images into specific categories. However, this rigid classification often acts as a bottleneck, restricting the generator's diversity and leading back to mode collapse.
Methodology
Text-SeGAN addresses these limitations through two key innovations: Semantic Relevance Regression and Curriculum-based Negative Sampling.
Semantic Relevance Regression
Unlike traditional Conditional GANs (CGANs) or Twin Auxiliary Classifier GANs (TAC-GANs) that use discrete class labels, Text-SeGAN's discriminator incorporates a regressor. This regressor outputs a fractional value between 0 and 1, representing the semantic alignment between the image and the text description. This allows the model to learn a continuous "relevance" manifold, facilitating a more natural one-to-many mapping.
Easy-to-Hard Negative Sampling
To refine the discriminator's understanding of semantic relevance, the authors employ a curriculum learning strategy for negative sampling. Training begins with "easy negatives"—images whose descriptions are semantically distant from the target text. As training progresses, "hard negatives"—images that are semantically similar but technically incorrect—are introduced. This gradual increase in difficulty prevents early collapse and forces the model to capture fine-grained semantic details.
Experimental Results
The model was evaluated on the Oxford-102 flower dataset, consisting of 8,189 images across 102 species. Text-SeGAN achieved an Inception Score (IS) of 4.03, a substantial improvement over GAN-INT-CLS (2.66) and TAC-GAN (3.45).
As shown in the qualitative analysis, while baseline models often produce nearly identical images for a given prompt, Text-SeGAN generates flowers with varied shapes, petal counts, and orientations while maintaining strict adherence to the text's semantic features. Ablation studies confirmed that the curriculum sampling strategy alone boosted the Inception Score from 3.65 to 4.03.
Conclusion
Text-SeGAN demonstrates that moving away from rigid class labels toward a continuous semantic relevance space significantly enhances the performance of text-to-image GANs. The combination of relevance regression and curriculum-based negative sampling effectively mitigates mode collapse. Future research will explore scaling this framework to higher resolutions and more complex, multi-object datasets.
Summary updated