Language-agnostic Semantic Consistent Text-to-Image Generation

SeongJun Jung, Woo Suk Choi, Seongho Choi, Byoung-Tak ZhangACL 2022

TL;DR

This paper introduces LaSC-GAN, a novel approach to text-to-image generation that tackles the challenge of generating semantically consistent images across multiple languages, even with limited data for some. By employing a language-agnostic text encoder and a Siamese-structured training approach, LaSC-GAN achieves comparable image quality to high-resource languages and ensures semantic consistency between images generated from texts with the same meaning in different languages.

What Problem Does This Paper Address?

Generating realistic images from text descriptions has seen significant advancements with Generative Adversarial Networks (GANs). However, most existing models are limited to a single language, typically English. This paper addresses two key challenges in extending this capability to multilingual text-to-image generation:

  1. Language Imbalance in Datasets: The vast majority of text-image paired datasets are in English. This scarcity of data for other languages makes it difficult to train robust multilingual models.
  2. Semantic Consistency Across Languages: Even if models can generate images for different languages, ensuring that captions with the same meaning but in different languages produce semantically consistent images is a significant hurdle. Existing methods struggle to capture this cross-lingual semantic commonality, potentially leading to different visual interpretations for equivalent textual descriptions.

Methods

The proposed Language-agnostic Semantic Consistent Generative Adversarial Network (LaSC-GAN) tackles these problems through two main components:

  1. Language-Agnostic Text Encoder: This component is designed to generate text embeddings that can be used by the generator regardless of the input language. It utilizes a pre-trained LaBSE (Language-agnostic BERT Sentence Embedding) model for language adaptation and a bi-directional LSTM for visual adaptation. This allows the model to leverage knowledge from high-resource languages to generate images for low-resource languages.
  2. Hierarchical Generator with Siamese Structure: The generator builds upon existing hierarchical GAN architectures (Xu et al., 2018) to generate images at increasing resolutions. Crucially, for multilingual training (Stage 2), a Siamese structure is employed. This involves training two model branches simultaneously on different language pairs (e.g., English-Chinese, English-Korean). By using a contrastive loss based on the visual features from the discriminator of these two branches, the model learns to extract and enforce semantic consistency between images generated from texts with the same meaning across different languages.

The training strategy is divided into two stages:

  • Stage 1: The model is trained using only a high-resource language dataset (COCO-EN) with a DAMSM loss, similar to previous work.
  • Stage 2: The model is further trained using the Siamese mechanism on multilingual datasets, incorporating contrastive loss to enforce semantic consistency.

Datasets

The paper utilizes the following datasets:

  • MS-COCO (COCO-EN): Used for Stage 1 training, this is a standard English text-image dataset.
  • Multilingual versions of COCO-EN:
    • COCO-CN: A dataset with manually translated Chinese descriptions for a subset of COCO images.
  • COCO-KO: A dataset with machine-translated Korean descriptions for COCO-EN.

For Stage 2, subsets of data that overlap across language pairs (EN-CN, EN-KO) are used for training. The models are then evaluated on the validation sets of COCO-CN and COCO-KO.

Key Insights and Ideas

  • Zero-Shot Multilingual Generation: The language-agnostic text encoder is a powerful concept, enabling the model to generate images for languages it hasn't explicitly seen during Stage 1 training. This is a significant step towards truly universal text-to-image models.
  • Siamese Structure for Semantic Alignment: The use of a Siamese network in Stage 2 is key to explicitly learning and enforcing semantic consistency. By comparing visual features from parallel language branches, the model is guided to produce similar visual representations for semantically equivalent text.
  • Bridging the Gap for Low-Resource Languages: The paper demonstrates that by leveraging high-resource language data and a language-agnostic encoder, comparable image quality can be achieved for low-resource languages, mitigating the data imbalance problem.

Key Results

  • Comparable Image Quality: Experiments show that the LaSC-GAN model, even when trained on low-resource languages (or in a zero-shot setting for Stage 1), achieves Inception Scores (IS) and Fréchet Inception Distances (FID) that are not significantly worse than those achieved with the high-resource English dataset.
  • Semantic Consistency Demonstrated: Qualitative results (Figures 3 and 4) and quantitative metrics (Table 2, showing reduced FID between language pairs after Stage 2) indicate that the Siamese mechanism effectively extracts semantic commonalities, leading to more consistent image generation across languages.
  • Zero-Shot Capabilities: Figure 2 showcases impressive zero-shot generation for languages like Thai (TH) and Nepali (NE) after Stage 1 training solely on English.

Limitations and Future Work

  • Reliance on Pre-trained Models: The language-agnostic text encoder relies on pre-trained models like LaBSE. While effective, the performance is inherently tied to the capabilities of these pre-trained embeddings.
  • Dataset Size: While the paper addresses language imbalance, the multilingual datasets used (especially COCO-CN and COCO-KO) are still relatively smaller than the original COCO-EN.
  • Future Work: The authors suggest that their method can be extended to more languages and potentially more complex multimodal tasks. Further exploration into more sophisticated language-agnostic encoders and Siamese training strategies could yield even better results.

Summary updated

BibTeX
@inproceedings{jung2022language,
  author = {SeongJun Jung and Woo Suk Choi and Seongho Choi and Byoung-Tak Zhang},
  title = {{Language-agnostic Semantic Consistent Text-to-Image Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure