The Challenges of Image Generation Models in Generating Multi-Component Images

Tham Yik Foong, Shashank Kotyan, Poyuan Mao, Danilo Vasconcellos VargasarXiv 2023 · DOI · arXiv

Introduction

While modern text-to-image models can generate stunning single objects, they frequently struggle to juggle multiple components in a single scene. When asked for a simple subject like "a cat," state-of-the-art models like Stable Diffusion excel. However, as prompt complexity increases—for example, requesting "a crab, a macaw, a steel drum, and a red butterfly"—models often suffer from "prompt neglect," dropping one or more items entirely.

Comparison of image generation models on a multi-component prompt. Stable Diffusion, AttnGAN, and DALL-E mini show varying degrees of success in including all requested objects.

This research identifies a critical "bottleneck" in generative AI: as prompt complexity rises, both the accuracy of object inclusion and the overall image quality decline sharply. Standard metrics like FID (Fréchet Inception Distance) or Inception Score (IS) measure realism and diversity but fail to explicitly count whether a model followed instructions to include every requested item. To address this, the authors introduce the Components Inclusion Score (CIS) and the Multi-Component Image Dataset (MCID).

Quantifying Success: The Components Inclusion Score (CIS)

To move beyond general similarity metrics like CLIP Score, the researchers proposed the CIS. This quantitative metric uses a CLIP-based lookup table to determine exactly how many of the $K$ requested components are present in a generated image.

The CIS evaluation pipeline. It uses a lookup table of possible component combinations and a CLIP-based model to calculate the probability of successfully rendered items.

The CIS calculates the probability of various combinations of requested objects to find the most likely count of successfully rendered items. By comparing the generated image against a lookup table $L(V)$ of all possible subsets of the prompt components, the metric provides a granular view of model performance as complexity scales.

The Multi-Component Image Dataset (MCID)

To help models learn complex compositions, the authors introduced the MCID, a benchmark containing 1.28 million images. The dataset includes 160k images for each count of 1 through 8 components. These images were constructed by combining existing ImageNet samples into single frames, providing a "ground truth" for multi-object scenes that models can use for training and evaluation.

Key Insights and Results

The study compared four major models: DALL-E mini, GLIDE, AttnGAN, and Stable Diffusion V2.

The "Complexity Tax"

The researchers identified a consistent performance drop as prompts grew more crowded. For every additional component added, the CIS dropped by an average of 8.53%. Furthermore, a "quality tax" was observed: as component counts increased, the Inception Score dropped by nearly 16%, and the FID increased by nearly 10%, suggesting that overcrowding an image degrades the visual quality of individual parts.

Model Performance and Fine-Tuning

  • Stable Diffusion V2 proved the most robust, maintaining a CIS of 0.674 even with 8 components (averaging 5 out of 8 items rendered).
  • Fine-tuning with LoRA: By training Stable Diffusion on the MCID dataset using Low-Rank Adaptation (LoRA), the authors boosted its overall CIS by 4.55%. Interestingly, the fine-tuned model began creating "borders" or distinct cells within images to organize the canvas and ensure every requested item had its own space.

Model Bias

Failures were not random. Models showed significant bias, often correctly generating common objects like "carved pumpkins" while ignoring less frequent ones like "chiffonier." This highlights how training data skew impacts a model's ability to handle diverse, multi-component prompts.

Limitations and Future Directions

While the MCID and CIS provide a path forward, challenges remain. The current MCID images, being stitched together, lack natural interactions between objects (e.g., a bird sitting on a drum). Additionally, the CIS relies on CLIP, which itself may struggle with counting very high numbers of distinct objects.

Future research should focus on spatial reasoning—ensuring objects are not just present, but interacting correctly (e.g., "the cat is under the table"). The authors also suggest exploring iterative inpainting as a method to "add" objects one by one to a scene, potentially overcoming the simultaneous generation bottleneck.

Summary updated

Abstract

Recent advances in text-to-image generators have led to substantial capabilities in image generation. However, the complexity of prompts acts as a bottleneck in the quality of images generated. A particular under-explored facet is the ability of generative models to create high-quality images comprising multiple components given as a prior. In this paper, we propose and validate a metric called Components Inclusion Score (CIS) to evaluate the extent to which a model can correctly generate multiple components. Our results reveal that the evaluated models struggle to incorporate all the visual elements from prompts with multiple components (8.53% drop in CIS per component for all evaluated models). We also identify a significant decline in the quality of the images and context awareness within an image as the number of components increased (15.91% decrease in inception Score and 9.62% increase in Frechet Inception Distance). To remedy this issue, we fine-tuned Stable Diffusion V2 on a custom-created test dataset with multiple components, outperforming its vanilla counterpart. To conclude, these findings reveal a critical limitation in existing text-to-image generators, shedding light on the challenge of generating multiple components within a single image using a complex prompt.
BibTeX
@misc{foong2023challenges,
  author = {Tham Yik Foong and Shashank Kotyan and Poyuan Mao and Danilo Vasconcellos Vargas},
  title = {{The Challenges of Image Generation Models in Generating Multi-Component Images}},
  year = {2023},
  eprint = {2311.13620},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.13620},
}
Figure