Text-to-Image Synthesis Based on Machine Generated Captions

Marco Menardi, Alex Falcon, Saida S. Mohamed, Lorenzo Seidenari, Giuseppe Serra, Alberto Del Bimbo, Carlo Tasso

Introduction

Text-to-image synthesis represents a significant challenge in artificial intelligence, aiming to generate realistic visual content from natural language descriptions. While the potential applications are vast—ranging from creative tools to accessibility enhancements—the primary bottleneck remains the scarcity of high-quality, paired image-text datasets. While millions of uncaptioned images are available online, human-annotated datasets like MSCOCO are relatively small and expensive to produce.

This paper proposes a "bootstrapping" approach to overcome this data scarcity. The core idea is to utilize an existing image captioning model to automatically generate descriptions for massive uncaptioned datasets, which are then used to train a conditional Generative Adversarial Network (GAN).

Conceptual overview of the bootstrapping pipeline for text-to-image synthesis.

Methodology

The proposed pipeline consists of three primary components:

  1. Image Captioning Module: A captioning system based on FC models is trained on the MSCOCO dataset. To improve performance and avoid "exposure bias," the model is optimized using Self-Critical Sequence Training (SCST), a reinforcement learning technique that directly optimizes for evaluation metrics like CIDEr.
  2. The Bridge: This module generates descriptions for a target domain of uncaptioned images (e.g., the LSUN-bedroom dataset). These synthetic captions are then converted into text embeddings using Skip-Thought vectors.
  3. GAN Module: The system employs the StackGAN-v2 architecture, which generates images in multiple stages of increasing resolution, conditioned on the Skip-Thought embeddings of the machine-generated captions.

Datasets and Training

The experiments utilized two primary datasets:

  • MSCOCO: Comprising 120,000 images with human-written captions, used for the initial training of the captioning module.
  • LSUN-bedroom: A target domain containing approximately 3 million uncaptioned bedroom images. A subset of 120,000 images was used to test the synthesis pipeline.

The Diversity Gap and Specificity Trap

A critical finding of this research is the "specificity trap." While the captioning module successfully transferred its general understanding from MSCOCO to the bedroom domain, the resulting captions were often overly generic. For instance, instead of describing unique features like a "vintage lamp" or "blue curtains," the AI frequently produced repetitive phrases such as "this bedroom looks very old-fashioned."

This lack of diversity creates a "noise vs. signal" problem. Because the GAN receives nearly identical text inputs for visually distinct images, it begins to treat the conditioning text as noise. Consequently, the model struggles to learn meaningful mappings between specific words and visual features, even if the overall image quality remains high.

Results and Discussion

Qualitatively, the GAN successfully learned the visual distribution of the target domain, producing realistic bedroom images. The visual fidelity was found to be comparable to models trained on much larger datasets, suggesting that the volume of data is less of a hurdle than the quality of the conditioning labels.

A sample bedroom image generated by the GAN conditioned on machine-generated text.

Quantitatively, the authors noted that standard metrics like the Inception Score are often unreliable for specialized domains like LSUN. However, the qualitative "conditioning failure" remains the primary takeaway: while the images look like bedrooms, they do not consistently reflect the specific nuances of the input text due to the generic nature of the synthetic captions.

Conclusion and Future Work

This study demonstrates that while machine-generated captions can facilitate the training of image generators on uncaptioned data, the diversity of those captions is a limiting factor. Future research should focus on enhancing caption variety, perhaps through GAN-based captioning or hybrid human-in-the-loop approaches, to better bridge the gap between synthetic labels and realistic image synthesis.

Summary updated

Abstract

Text to Image Synthesis refers to the process of automatic generation of a photo-realistic image starting from a given text and is revolutionizing many real-world applications. In order to perform such process it is necessary to exploit datasets containing captioned images, meaning that each image is associated with one (or more) captions describing it. Despite the abundance of uncaptioned images datasets, the number of captioned datasets is limited. To address this issue, in this paper we propose an approach capable of generating images starting from a given text using conditional GANs trained on uncaptioned images dataset. In particular, uncaptioned images are fed to an Image Captioning Module to generate the descriptions. Then, the GAN Module is trained on both the input image and the machine-generated caption. To evaluate the results, the performance of our solution is compared with the results obtained by the unconditional GAN. For the experiments, we chose to use the uncaptioned dataset LSUN bedroom. The results obtained in our study are preliminary but still promising.
BibTeX
@inproceedings{menardi2020text,
  author = {Marco Menardi and Alex Falcon and Saida S. Mohamed and Lorenzo Seidenari and Giuseppe Serra and Alberto Del Bimbo and Carlo Tasso},
  title = {{Text-to-Image Synthesis Based on Machine Generated Captions}},
  booktitle = {IRCDL},
  year = {2020},
  doi = {10.1007/978-3-030-39905-4\_7},
}
Figure