Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds

Shuangqi Li, Hieu Le, Jingyi Xu, Mathieu Salzmann

Executive Summary

Researchers have discovered that the "random" noise used to initialize image generation in diffusion models isn't entirely neutral. Instead, certain random seeds act as "invisible templates" that naturally favor specific object counts and spatial layouts. By mining these reliable seeds and fine-tuning models on their own most successful outputs, the authors achieved massive improvements in compositional accuracy—including up to a 60% boost in spatial relations—without requiring any manual human annotation.

The Compositional Generation Problem

Current text-to-image (T2I) diffusion models, such as Stable Diffusion, excel at artistic generation but often struggle with basic counting and spatial reasoning. If prompted for "four unicorns," a model might generate three or five. If asked for a "penguin on the right of a bowl," it frequently flips the positions or merges the concepts haphazardly.

Comparison of baseline Stable Diffusion and the proposed method on compositional prompts.

This challenge is known as the compositional generation problem. While previous research has attempted to fix this by forcing models to follow strict layouts or using Large Language Models (LLMs) to plan bounding boxes, these methods are often slow, labor-intensive, or restrict the model's creative flexibility. This paper asks a fundamental question: Can we improve a model's compositional ability simply by understanding and leveraging the initial noise it starts with?

Methodology: Mining and Fine-Tuning

The authors introduce a three-step framework to turn "lucky" noise into a permanent skill for the model:

  1. Reliable Seed Mining: The team generated thousands of images using different random seeds. They then employed a Vision-Language Model (VLM), specifically CogVLM2, to act as an automated judge. The VLM verified whether the generated image actually contained the requested number of objects or the correct spatial relationship. Seeds that consistently produced correct results across various prompts were labeled "reliable."
  2. Inference-time Sampling: The most straightforward application of this discovery is to use these "golden seeds" during generation. For instance, if a user wants four objects, the system can initialize the generation with a seed that the mining process identified as favoring a "four-grid" layout.
  3. Fine-Tuning on Self-Generated Data: To improve the model's baseline performance regardless of the seed used, the researchers fine-tuned it on a curated dataset of its own successful "reliable" images. Crucially, they only updated the Query (Q) and Key (K) projection layers within the attention modules. This surgical fine-tuning preserves the model's overall image quality while specifically sharpening its understanding of how text relates to object placement.

The Comp90 Benchmark

To rigorously test their method, the authors introduced Comp90, a new benchmark specifically designed for compositional evaluation:

  • Categories: 90 distinct object types (animals, food, common objects).
  • Backgrounds: 12 hand-crafted settings (e.g., "on a rocky alien planet").
  • Prompts: 3,000 prompts for numerical composition (counting 2–6) and 3,200 for spatial relations (top, under, left, right).
  • Split: 60 categories were used for training/mining, and 30 "unseen" categories were reserved for testing to ensure the model learned the concept of counting and placement, rather than just memorizing how to draw specific training objects.

Key Insights: Seeds as Layouts

The most striking finding of this research is that initial noise patterns are not just random static; they actively guide the model toward specific "camera angles" and compositions.

Images generated with the same seed share similar layouts and object counts, regardless of the prompt.

As seen above, a specific seed (like seed = 50) consistently produces a two-object layout slightly offset from the center, whether the prompt asks for apples, oranges, coins, or umbrellas. Another seed (seed = 23) consistently scatters three objects.

Furthermore, the authors identified a "Four-Grid" Phenomenon. Certain seeds naturally partition the image space into quadrants. These seeds are significantly more reliable for prompts requiring exactly four objects because they provide a natural "slot" for each item. By using a VLM to "rectify" prompts (e.g., if a model draws five apples for a "four apples" prompt, the VLM re-labels that image as "five apples" for training), the model learns from its mistakes rather than being penalized for them.

Results and Performance

The proposed method was tested on Stable Diffusion 2.1 and PixArt-α, demonstrating dramatic improvements:

  • Numerical Accuracy: Stable Diffusion achieved a 29.3% relative increase in counting accuracy, while PixArt-α saw a 19.5% increase.
  • Spatial Accuracy: The gains in spatial composition were even more substantial, with a 60.7% relative increase for Stable Diffusion (moving from 17.8% to 36.6% accuracy) and a 21.1% increase for PixArt-α.
  • Generalization: The fine-tuned models improved even on "out-of-scope" quantities (like asking for 7 or 8 objects) and multiple-category prompts (e.g., "three tigers and two penguins"), proving they learned general compositional rules.
  • Aesthetics: Unlike other layout-control methods that can produce robotic or "glitchy" images, this self-supervised fine-tuning method maintained, and sometimes even improved, the aesthetic quality of the generated images.

Limitations and Future Work

While highly effective, the approach has a few limitations. First, there is a trade-off between diversity and accuracy: relying exclusively on a small set of "reliable" seeds can lead to repetitive layouts. The authors mitigate this through fine-tuning, but the tension between strict prompt adherence and creative variety remains. Second, the system's success is bottlenecked by the VLM's accuracy; if the VLM struggles to count or understand complex spatial relations, those errors will be baked into the fine-tuned T2I model. Finally, while the model handles simple "A on top of B" relationships well, highly complex scenes with numerous interacting objects remain a challenge for future iterations of seed mining.

Summary updated

Abstract

Text-to-image diffusion models have demonstrated remarkable capability in generating realistic images from arbitrary text prompts. However, they often produce inconsistent results for compositional prompts such as "two dogs" or "a penguin on the right of a bowl". Understanding these inconsistencies is crucial for reliable image generation. In this paper, we highlight the significant role of initial noise in these inconsistencies, where certain noise patterns are more reliable for compositional prompts than others. Our analyses reveal that different initial random seeds tend to guide the model to place objects in distinct image areas, potentially adhering to specific patterns of camera angles and image composition associated with the seed. To improve the model's compositional ability, we propose a method for mining these reliable cases, resulting in a curated training set of generated images without requiring any manual annotation. By fine-tuning text-to-image models on these generated images, we significantly enhance their compositional capabilities. For numerical composition, we observe relative increases of 29.3\% and 19.5\% for Stable Diffusion and PixArt-$\alpha$, respectively. Spatial composition sees even larger gains, with 60.7\% for Stable Diffusion and 21.1\% for PixArt-$\alpha$.
BibTeX
@inproceedings{li2025enhancing,
  author = {Shuangqi Li and Hieu Le and Jingyi Xu and Mathieu Salzmann},
  title = {{Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=5BSlakturs},
}
Figure