CRD-CGAN: Category-Consistent and Relativistic Constraints for Diverse Text-to-Image Generation

Tao Hu, Chengjiang Long, Chunxia Xiao

Introduction

Text-to-image generation using Generative Adversarial Networks (GANs) has made significant strides, yet two major hurdles persist: the "diversity gap" and the "category consistency issue." The diversity gap refers to the tendency of models to produce nearly identical images for a single prompt, a phenomenon known as mode collapse. Category consistency involves the model's failure to adhere to the specific visual traits of a real-world category. For instance, a model might generate a realistic bird that lacks the specific color patterns defining its species, focusing on local text alignment while ignoring global visual concepts.

Methodology

The Category-Consistent and Relativistic Diverse Conditional GAN (CRD-CGAN) addresses these issues through a multi-faceted architecture consisting of three primary pillars.

Multi-Generator Framework (D-CGAN)

To combat mode collapse, CRD-CGAN utilizes $K$ generators sharing a single discriminator. Each generator receives a unique noise vector. A diversity loss function is applied, which maximizes the ratio between the distance in the generated image space and the distance in the latent noise space, forcing the generators to produce distinct visual outputs for the same text prompt simultaneously.

Relativistic Conditional Loss

Traditional GAN discriminators output an absolute probability of an image being real or fake. CRD-CGAN adopts a relativistic approach, estimating the probability that a real image is more realistic than the average synthetic image, and vice versa. This relative comparison provides more stable gradients during training and encourages the model to capture finer, more realistic details.

Category-Consistent Loss

The core innovation of CRD-CGAN is the category-consistency loss. By extracting features from both real and synthetic images and calculating their cosine similarity, the model ensures that synthetic images remain anchored to their ground-truth categories. This acts as a visual "anchor," preventing the generation of "generic" objects and ensuring that species-specific traits—such as the exact shade of black for an American Crow—are preserved.

Experimental Results

The model was evaluated on the CUB (Birds-200-2011), Oxford-102 Flowers, and MS COCO 2014 datasets, demonstrating superior performance across quality and diversity metrics.

Quantitative Performance

CRD-CGAN showed significant improvements in both diversity and quality:

  • Diversity (LPIPS): On the CUB dataset, it achieved an LPIPS score of 39.91%, notably higher than AttnGAN (35.26%) and StackGAN++ (31.37%).
  • Quality (FID): The model reached a competitive FID of 24.59 on CUB and a state-of-the-art 40.75 on the Oxford Flower dataset.
  • Human Evaluation: In user studies, CRD-CGAN images were preferred for their realism significantly more often than baseline models (e.g., 33.24% preference vs. 12.38% for AttnGAN on birds).

Qualitative Insights

Visual results demonstrate that CRD-CGAN is highly sensitive to semantic nuances in text prompts. For example, changing a description from "white belly" to "yellow belly" results in accurate color shifts while the overall category-specific structure of the bird remains intact. This effectively solves the "Grey Crow" problem where previous models would often ignore category-specific colors in favor of generic bird features.

Conclusion and Future Work

CRD-CGAN effectively bridges the gap between text-to-image alignment and category-specific realism. While highly successful on fine-grained datasets like birds and flowers, complex multi-object scenes in datasets like MS COCO remain a challenge for GAN-based methods. Future research could extend category-consistency constraints to individual objects within multi-object scenes and optimize the computational efficiency of the multi-generator architecture for real-time deployment.

Summary updated

Abstract

Generating photo-realistic images from a text description is a challenging problem in computer vision. Previous works have shown promising performance to generate synthetic images conditional on text by Generative Adversarial Networks (GANs). In this paper, we focus on the category-consistent and relativistic diverse constraints to optimize the diversity of synthetic images. Based on those constraints, a category-consistent and relativistic diverse conditional GAN (CRD-CGAN) is proposed to synthesize $K$ photo-realistic images simultaneously. We use the attention loss and diversity loss to improve the sensitivity of the GAN to word attention and noises. Then, we employ the relativistic conditional loss to estimate the probability of relatively real or fake for synthetic images, which can improve the performance of basic conditional loss. Finally, we introduce a category-consistent loss to alleviate the over-category issues between K synthetic images. We evaluate our approach using the Birds-200-2011, Oxford-102 flower and MSCOCO 2014 datasets, and the extensive experiments demonstrate superiority of the proposed method in comparison with state-of-the-art methods in terms of photorealistic and diversity of the generated synthetic images.
BibTeX
@article{hu2021crd,
  author = {Tao Hu and Chengjiang Long and Chunxia Xiao},
  title = {{CRD-CGAN: Category-Consistent and Relativistic Constraints for Diverse Text-to-Image Generation}},
  journal = {Frontiers of Computer Science},
  year = {2021},
  doi = {10.1007/s11704-022-2385-x},
}
Figure