Multilingual Conceptual Coverage in Text-to-Image Models

Michael Saxon, William Yang Wang

Abstract: We propose "Conceptual Coverage Across Languages" (CoCo-CroLa), a technique for benchmarking the degree to which any generative text-to-image system provides multilingual parity to its training language in terms of tangible nouns. For each model we can assess "conceptual coverage" of a given target language relative to a source language by comparing the population of images generated for a series of tangible nouns in the source language to the population of images generated for each noun under translation in the target language. This technique allows us to estimate how well-suited a model is to a target language as well as identify model-specific weaknesses, spurious correlations, and biases without a-priori assumptions. We demonstrate how it can be used to benchmark T2I models in terms of multilinguality, and how despite its simplicity it is a good proxy for impressive generalization.

Introduction

State-of-the-art text-to-image (T2I) models, such as Stable Diffusion and DALL-E 2, have demonstrated an extraordinary ability to synthesize complex visual scenes from natural language. However, these models are predominantly trained on English-centric datasets, leaving their performance in other languages largely unquantified. Evaluating multilingual T2I systems is inherently difficult because the creative prompt space is infinite and subjective.

This paper introduces CoCo-CroLa (Conceptual Coverage Across Languages), a benchmark designed to measure a model's foundational understanding of basic nouns across different languages. We argue that "concept possession"—the ability to reliably generate a simple image of a tangible noun—is a prerequisite for complex compositional creativity. If a model cannot generate a "bird" when prompted in Spanish, it cannot be expected to generate "a bird using a keyboard in the snow" in that same language.

The CoCo-CroLa Methodology

CoCo-CroLa assesses a model’s possession of a concept using minimal eliciting prompts (e.g., "a photograph of [word]"). By stripping away complex syntax, we isolate the model's ability to map a specific lexical token to a visual representation.

Evaluation Metrics

To quantify conceptual coverage, we utilize four primary metrics:

  1. Correctness (Cross-consistency, $X_c$): Measures the similarity between images generated in the target language (e.g., Indonesian) and those generated in the source language (English).
  2. Self-consistency ($S_c$): Evaluates whether multiple generations of the same prompt yield visually similar results, indicating the model is not merely "guessing."
  3. Distinctiveness ($D_t$): Ensures that images for different nouns (e.g., "dog" vs. "airplane") are visually distinct, helping identify "generic collisions" where a model defaults to a standard output for unknown terms.
  4. Word-Image Similarity ($W_c$): A direct measure using CLIP embeddings to evaluate how well the generated image aligns with the text prompt.

Dataset and Scope

The CoCo-CroLa v1.0 benchmark consists of 193 tangible concepts (e.g., "pizza," "mountain," "bicycle") translated across seven languages: English (EN), Spanish (ES), German (DE), Chinese (ZH), Japanese (JP), Hebrew (HE), and Indonesian (ID). We evaluated nine prominent models, including DALL-E Mini/Mega, various iterations of Stable Diffusion, CogView 2, DALL-E 2, and AltDiffusion.

Key Insights

Concept Possession as a Prerequisite

Our findings confirm that concept possession is upstream of compositional ability. Models that fail the CoCo-CroLa test for a specific noun invariably fail more complex prompts involving that noun. This establishes CoCo-CroLa as an efficient proxy for broader multilingual generalization.

The Impact of Script Sharing

Performance is heavily influenced by script. Models trained primarily on English data perform significantly better on languages using the Latin script (Spanish, German, Indonesian). Conversely, languages with unique scripts, such as Hebrew, often serve as outliers for failure in English-centric models.

Failure Modes: Specific vs. Generic Collisions

When a model lacks a concept in a target language, it typically fails in one of two ways:

  • Specific Collision: The model consistently maps a word to the wrong concept (e.g., Japanese for "dog" producing a "human").
  • Generic Collision: The model produces a "safe" default image—such as a generic landscape—regardless of the specific noun provided.

The Success of AltDiffusion

AltDiffusion, which utilized specific multilingual contrastive training, significantly outperformed other models in language parity. While it saw a slight decrease in English performance compared to its base Stable Diffusion model, its ability to bridge the gap between languages suggests that intentional multilingual alignment is a necessary path forward for global T2I systems.

Results and Thresholds

Our benchmarking revealed that English dominance remains the norm; for almost all models (excluding the Chinese-trained CogView 2), English performance was vastly superior. We identified specific failure thresholds: a Correctness ($X_c$) score below 0.5 or a Word-Similarity ($W_c$) score below 25 almost always indicates a total failure to generate that concept in complex scenes.

Limitations and Future Work

While CoCo-CroLa provides a robust baseline, it is not without limitations. The metrics rely on CLIP, which carries its own English-centric biases. Furthermore, "tangible nouns" can have cultural nuances that simple translation may miss (e.g., the visual representation of "food" varies by region).

Future research should expand this benchmark to include verbs and adjectives to test "compositional coverage" and explore the use of more robust, purpose-trained multilingual classifiers to move beyond the limitations of CLIP.

Conclusion

CoCo-CroLa offers a simple yet powerful framework for auditing the multilingual capabilities of T2I models. By focusing on the foundational building blocks of language—tangible nouns—we can identify model-specific weaknesses and biases, providing a roadmap for creating truly global generative AI.

Summary updated

Abstract

We propose "Conceptual Coverage Across Languages" (CoCo-CroLa), a technique for benchmarking the degree to which any generative text-to-image system provides multilingual parity to its training language in terms of tangible nouns. For each model we can assess "conceptual coverage" of a given target language relative to a source language by comparing the population of images generated for a series of tangible nouns in the source language to the population of images generated for each noun under translation in the target language. This technique allows us to estimate how well-suited a model is to a target language as well as identify model-specific weaknesses, spurious correlations, and biases without a-priori assumptions. We demonstrate how it can be used to benchmark T2I models in terms of multilinguality, and how despite its simplicity it is a good proxy for impressive generalization.
BibTeX
@inproceedings{saxon2023multilingual,
  author = {Michael Saxon and William Yang Wang},
  title = {{Multilingual Conceptual Coverage in Text-to-Image Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=9mCa_iIGbs},
}
Figure