Recurrent Affine Transformation for Text-to-Image Synthesis

Senmao Ye, Huan Wang, Mingkui Tan, Fei LiuIEEE Trans. Multimedia 2023 · DOI

TL;DR

This paper introduces a novel "Recurrent Affine Transformation" (RAT) for text-to-image synthesis. By connecting existing affine transformation blocks with a recurrent neural network (RNN), RAT ensures consistent text information fusion across different layers, leading to more coherent and higher-quality generated images. Additionally, a refined spatial attention mechanism in the discriminator improves the semantic alignment between text and image.

What Problem Does This Paper Address?

Generating realistic images from text descriptions is a challenging task, primarily due to the difficulty of effectively integrating textual information into the image synthesis process. Current methods often rely on independent "fusion blocks" (like Conditional Batch Normalization or Instance Normalization) at various layers of the generator. However, these blocks operate in isolation, leading to inconsistencies and making the training process more complex. The paper aims to solve this by creating a more unified and globally aware fusion mechanism.

Methods

The core innovation is the Recurrent Affine Transformation (RAT). Instead of treating affine transformations (which adapt image features based on text) as isolated modules, RAT connects them using a Recurrent Neural Network (RNN), specifically an LSTM variant. This recurrent connection allows the model to learn long-term dependencies between these fusion blocks, ensuring that text information is integrated consistently across all layers. This approach also simplifies training due to skip connections.

For the discriminator, the authors introduce a spatial attention mechanism. This allows the discriminator to focus on image regions that are most relevant to the text description. Crucially, they replace the standard softmax function with a soft threshold function to prevent attention probabilities from collapsing to near zero, thereby improving training stability and avoiding model collapse.

Datasets

The proposed model was evaluated on three standard datasets:

  • CUB (Caltech-UCSD Birds-200-2011): Contains 11,788 images of birds across 200 categories.
  • Oxford-102: Features 8,189 images of flowers across 102 categories.
  • MS COCO: A large-scale dataset with 123,287 images and 5 sentence annotations per image.

Key Insights and Ideas

  • Global Consistency is Key: The paper highlights that isolated fusion blocks, while effective individually, fail to capture the global context of text-image relationships. Connecting them via an RNN (RAT) creates a more holistic fusion process.
  • RNNs for Feature Fusion: Applying RNNs to manage the flow of text information through affine transformations is a novel way to enforce consistency and reduce training complexity.
  • Soft Threshold for Stable Attention: The standard softmax function in attention mechanisms can lead to model instability in GANs. Using a soft threshold function is a simple yet effective modification to maintain stable gradients and prevent probability collapse.
  • Discriminator's Role in Text-Image Alignment: By incorporating spatial attention, the discriminator becomes more adept at identifying relevant image regions for a given text, which in turn guides the generator to produce more semantically accurate images.

Key Results

The proposed RAT-GAN achieved state-of-the-art results on the CUB and Oxford-102 datasets, demonstrating superior Inception Scores (IS) and lower Fréchet Inception Distances (FID). Specifically, on the CUB dataset, the model achieved an IS of 5.36 and an FID of 13.91. The authors also noted that their model performs particularly well on the more complex COCO dataset, outperforming other methods. Qualitative results showed generated images with clearer details and more relevant content compared to baselines like StackGAN and DF-GAN.

Limitations and Future Work

The authors acknowledge that while their spatial attention mechanism is more stable than vanilla softmax, it still relies on a soft threshold. Future work could explore applying RAT to other conditional image synthesis tasks. The paper also suggests that further investigation into the interactions within the RAT blocks could yield additional improvements.

Summary updated

Abstract

Text-to-image synthesis aims to generate realistic images conditioned on text descriptions. To fuse text information into synthesized images, conditional affine transformations (CATs), such as conditional batch normalization (CBN) and conditional instance normalization (CIN), are usually used to predict batch statistics of different layers. However, ordinary CAT blocks control the batch statistics independently disregarding the consistency among neighboring layers. To address the above issue, we propose a new fusion approach names recurrent affine transformation (RAT) for synthesizing images conditioned on text information. RAT connects all the CAT blocks with recurrent connections for explicitly fitting the temporal consistency between CAT blocks. To verify the effectiveness of RAT, we propose a novel visualization method to show how generative adversarial network (GAN) fuses conditional information. Our microscopic and macroscopic visualizations not only demonstrate the effectiveness of RAT but also turn out to be a useful perspective to analyze how GAN fuses conditional information. In addition, we propose a more stable spatial attention mechanism for the discriminator, which helps the text description to supervise the generator to synthesize more relevant image contents. Extensive experiments on the CUB, Oxford-102, and COCO datasets demonstrate the proposed model's superiority in comparison to state-of-the-art models. Our code is available on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/senmaoy/RAT-GAN</uri> .
BibTeX
@article{ye2023recurrent,
  author = {Senmao Ye and Huan Wang and Mingkui Tan and Fei Liu},
  title = {{Recurrent Affine Transformation for Text-to-Image Synthesis}},
  journal = {IEEE Transactions on Multimedia},
  year = {2023},
  doi = {10.1109/tmm.2023.3266607},
}
Figure