Multilingual Conceptual Coverage in Text-to-Image Models

Michael Saxon, William Yang Wang

Introduction

While generative text-to-image (T2I) models like Stable Diffusion and DALL-E 2 have achieved remarkable success in generating high-fidelity images from English prompts, their performance in other languages remains largely under-evaluated. Most current assessments rely on English-centric metrics of "subjective impressiveness," overlooking the fundamental question of whether these models possess basic conceptual knowledge across diverse languages. This paper introduces CoCo-CroLa (Conceptual Coverage Across Languages), a benchmark designed to evaluate how well T2I models understand basic nouns globally.

The CoCo-CroLa Benchmark

The researchers argue that "concept possession"—the ability to generate a simple image of a noun—is a mandatory prerequisite for a model's ability to handle complex, creative prompts. To measure this, they developed a four-step pipeline:

  1. Translation-Aligned Concept List: A curated set of 193 tangible nouns (e.g., "dog," "airplane," "pizza") translated across seven languages: English (EN), Spanish (ES), German (DE), Chinese (ZH), Japanese (JP), Hebrew (HE), and Indonesian (ID).
  2. Minimal Eliciting Prompts: Using simple templates like "a photograph of [word]" to isolate the model's understanding of the noun itself.
  3. Image Generation & Embedding: Generating 10 images per (concept, language) pair and extracting semantic features using a CLIP image encoder.
  4. Quantitative Metrics: Performance is measured through Self-consistency (Sc), Distinctiveness (Dt), and Correctness (Xc & Wc), which compares non-English outputs to English benchmarks and CLIP text embeddings.

Concept Possession and the "Collision" Phenomenon

A key insight from the study is that basic concept possession acts as a proxy for downstream creativity. If a model fails to generate a simple "dog" in Japanese, it will inevitably fail a complex prompt like "a dog wearing a hat on the moon" in that same language.

When models fail to grasp a concept in a specific language, they often exhibit "collisions." These fall into two categories:

  • Specific Collisions: The model consistently maps a word to the wrong visual concept. For example, a model might associate the Japanese word for "airplane" with a green landscape instead.
  • Generic Collisions: Common in lower-resource languages like Hebrew, models may default to generating generic, culturally-associated scenes (e.g., vaguely Middle Eastern outdoor settings) regardless of the specific prompt.
Figure 4: Visualizing conceptual coverage and collisions across models and languages for 'dog', 'airplane', and 'face'.

Key Findings and Model Performance

The benchmark revealed several critical trends in current T2I development:

  • AltDiffusion m9 emerged as the most robust model, achieving the highest average correctness across all tested languages due to its specific multilingual training.
  • Script Sharing: Performance is highly correlated among languages that share scripts (e.g., Latin-based EN, ES, DE, ID). If a model understands a concept in Spanish, it is highly likely to understand it in German.
  • The Hebrew Outlier: Hebrew (HE) proved to be the most challenging language for nearly all models, likely due to its unique script and relative under-representation in training datasets.
  • Monolingual Bias: Models like CogView 2, trained primarily on Chinese, excelled in ZH but performed significantly worse in English and other languages compared to their peers.

Conclusion and Future Work

CoCo-CroLa provides a scalable, objective framework for identifying model-specific weaknesses and biases. While the current version focuses on tangible nouns, the authors acknowledge limitations, such as the potential English-centric bias of the CLIP feature extractor. Future research will aim to expand the benchmark to include verbs, adjectives, and abstract concepts to better understand how models handle complex actions and relationships across the global linguistic landscape.

Summary updated

Abstract

We propose "Conceptual Coverage Across Languages" (CoCo-CroLa), a technique for benchmarking the degree to which any generative text-to-image system provides multilingual parity to its training language in terms of tangible nouns. For each model we can assess "conceptual coverage" of a given target language relative to a source language by comparing the population of images generated for a series of tangible nouns in the source language to the population of images generated for each noun under translation in the target language. This technique allows us to estimate how well-suited a model is to a target language as well as identify model-specific weaknesses, spurious correlations, and biases without a-priori assumptions. We demonstrate how it can be used to benchmark T2I models in terms of multilinguality, and how despite its simplicity it is a good proxy for impressive generalization.
BibTeX
@inproceedings{saxon2023multilingual,
  author = {Michael Saxon and William Yang Wang},
  title = {{Multilingual Conceptual Coverage in Text-to-Image Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=9mCa_iIGbs},
}

Loading related content...

Figure