A Picture is Worth a Thousand Words: Using Cross-Modal Transformers and Variational AutoEncoders to Generate Images from Text

Satyajit Kumar, Ehsan AdeliJournal of Student Research 2021 · DOI

Abstract

Text-to-image generation is one of the most complex problems in deep learning, where the application of Recurrent Neural Networks (RNNs) and Generative Adversarial Networks (GANs) has seen significant success. However, GANs prioritize the sharpness of the image rather than covering all the nuances of the text. Given that Transformers have recently outperformed RNNs and other neural network models in both the text and image spaces, we explored whether Transformer models can perform better in multi-modal tasks such as text-to-image synthesis. Our conclusion based on evaluating five Transformer-based models on the MS-COCO dataset showed that Transformers perform better but would need a significant amount of memory and compute resources.

Introduction

The quest to generate realistic images from textual descriptions—text-to-image synthesis—has historically been dominated by Generative Adversarial Networks (GANs) and Recurrent Neural Networks (RNNs). While these architectures have achieved impressive results, they possess inherent "blind spots." GANs are highly effective at producing sharp, high-resolution textures but often fail to capture the subtle semantic nuances of the input text. Conversely, RNNs, such as LSTMs, struggle to encode complex or unusual prompts because they cannot effectively model long-range dependencies and complex pairwise relationships.

This research investigates whether Transformer-based models, which have revolutionized Natural Language Processing (NLP), can be successfully paired with Variational AutoEncoders (VAEs) to bridge the gap between text tokens and image latent spaces, resulting in images that are both semantically accurate and visually coherent.

Methodology

We experimented with five distinct architectural approaches to evaluate the synergy between Transformers and VAEs:

  1. BERT Encoder with VQ-VAE: This model uses a pre-trained BERT model to generate text embeddings. By concatenating four layers to preserve rich linguistic information, the embeddings are fed through eight fully connected layers into a Vector Quantized-VAE (VQ-VAE) decoder.
  2. Siamese-BERT with VQ-VAE: This iteration improves the text encoder by training it in a "siamese-triplet" fashion. This approach forces the model to cluster similar sentences together in the embedding space while isolating dissimilar ones, providing a more structured input for the image generator.
  3. Encoder-Decoder Transformer with VQ-VAE 2: This model utilizes VQ-VAE 2, which employs a hierarchical latent space for improved image sharpness. The text is processed by a BERT-style encoder, while the decoder attempts to output 16 tokens representing the image's geometry.
  4. Transformer Encoder with VQ-VAE 2: A streamlined version of the above, using only a Transformer encoder and mean-pooling to generate a fixed-size vector for the VAE.
  5. Pure Image Transformer: A direct "words-to-pixels" approach that bypasses the VAE latent space entirely. It uses local self-attention to attempt to generate 32x32 pixel grids directly from text tokens.

Datasets

All models were trained and evaluated using the MS-COCO (Microsoft Common Objects in Context) dataset. This standard benchmark contains over 300,000 images, each accompanied by five descriptive captions, providing a robust foundation for multi-modal training.

Key Results

The quantitative evaluation utilized the Inception Score (IS), a standard metric that measures both the variety and the distinctness of generated images. While our experimental models did not reach the visual fidelity of state-of-the-art GANs due to hardware constraints, the Siamese-BERT approach showed a nearly 100% improvement over the basic BERT approach, demonstrating that structured embedding spaces are vital for multi-modal tasks.

Model Inception Score (MS-COCO)
BERT + VQ-VAE (Ours) 1.6 ± 0.01
Siamese BERT + VQ-VAE (Ours) 3.1 ± 0.01
GAN-INT-CLS 7.88 ± 0.07
DALL-E (Sample Size 128) 18.13 ± 0.44
DM-GAN (SOTA GAN) 30.49 ± 0.57
Sample image outputs generated by the experimental Transformer-VAE architectures.

Discussion: Insights and Challenges

The "Upsample" Information Leak

A critical discovery was the "information leak" occurring when using large fully connected layers to "stretch" a small text vector (size 768 or 4096) into a massive VQ-VAE latent vector (size 131,072). Analysis of Mean Squared Error (MSE) and cosine similarity revealed that after these layers, the model could no longer distinguish between similar and dissimilar sentence embeddings, leading to a loss of semantic precision.

The Inception Score Bias

We observed a philosophical divide in evaluation metrics. The Inception Score prioritizes image sharpness and texture. Because VAE-based models (including DALL-E) often produce "smoother" images than GANs, they frequently receive lower IS scores even when their semantic understanding of the text is superior. This suggests that IS may not be the perfect metric for evaluating text-to-image alignment.

The Necessity of Discretization

A major takeaway was that combining Transformers directly with continuous latent spaces often leads to model collapse. Successful models like DALL-E succeed because they discretize the image into tokens. This allows the Transformer to treat image generation as a translation task between two "languages": text tokens and image tokens.

Conclusion and Future Work

The primary limitation of this study was the significant compute and memory requirements. Training a "Pure Image Transformer" required downsampling images to 32x32 pixels, which sacrificed detail. To make Transformer-VAE architectures more viable, future research should focus on:

  1. Enlarging the Transformer vocabulary to better capture the VAE's latent space.
  2. Discretizing outputs into tokens rather than mapping to continuous spaces.
  3. Optimizing memory through local self-attention or efficient hierarchical VAEs to enable higher-resolution generation.

Summary updated

Abstract

Text-to-image generation is one of the most complex problems in deep learning, where the application of Recurrent Neural Networks (RNNs) and Generative Adversarial Networks (GANs) has seen significant success. However, GANs prioritize the sharpness of the image rather than covering all the nuances of the text. Given that Transformers have recently outperformed RNNs and other neural network models in both the text and image spaces, we explored whether Transformer models can perform better in multi-modal tasks such as text-to-image synthesis. Our conclusion based on evaluating five Transformer based models on the MS-COCO dataset showed that Transformers perform better but would need a significant amount of memory and compute resources.
BibTeX
@article{kumar2021picture,
  author = {Satyajit Kumar and Ehsan Adeli},
  title = {{A Picture is Worth a Thousand Words: Using Cross-Modal Transformers and Variational AutoEncoders to Generate Images from Text}},
  journal = {Journal of Student Research},
  year = {2021},
  doi = {10.47611/jsrhs.v10i4.2106},
}
Figure