Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds

Shuangqi Li, Hieu Le, Jingyi Xu, Mathieu Salzmann

Introduction

Modern text-to-image (T2I) models like Stable Diffusion and PixArt-α have revolutionized digital art, yet they frequently stumble on fundamental compositional tasks. A simple request for "four unicorns" might yield three, and spatial instructions like "a penguin on the right of a bowl" often result in incorrect positioning. This research identifies a critical, overlooked factor in these failures: the initial random noise, or "seed," used during the diffusion process.

Comparison of baseline Stable Diffusion 2.1 vs. the proposed method across numeracy and spatial relation tasks.

By treating the initial noise as a structural blueprint rather than just stochasticity, the authors demonstrate that "reliable seeds" can be mined to significantly boost compositional accuracy—achieving gains of up to 60% in spatial relations.

The Blueprint Hypothesis: Seeds as Structural Layouts

The core insight of this work is that initial random noise contains low-frequency information that dictates the "slots" where objects are likely to appear. Some seeds naturally favor a grid-like layout, while others might push the model toward a vertical stack. When a prompt's requirements (e.g., "four objects") conflict with a seed's inherent layout (e.g., a "two-slot" noise pattern), the model fails to follow instructions.

Crucially, this "reliability" is object-agnostic. A seed that successfully helps a model generate a specific count of objects in a specific arrangement is highly likely to succeed across different object categories and backgrounds. This suggests that reliability is a property of the noise structure itself.

Visualization of how specific seeds (Seed 50 vs. Seed 23) consistently influence object layout and count across different categories and backgrounds.

Methodology: From Mining to Fine-Tuning

The researchers developed a three-stage framework to leverage these structural blueprints:

  1. Reliable Seed Mining: Thousands of images were generated and evaluated by a high-performance Vision-Language Model (VLM), CogVLM2. This process identified seeds that consistently produced correct counts and spatial arrangements.
  2. Inference-time Selection: Using these pre-identified reliable seeds during standard generation provided an immediate 6% accuracy boost without any model training.
  3. Self-Correction & Fine-Tuning: To internalize these improvements, the authors created a high-quality synthetic dataset. If a model intended to create four objects but only created three, the training prompt was "rectified" to match the actual image content. The researchers then fine-tuned only the attention layers (specifically the Query and Key projections) of the T2I models. This targeted approach "bakes in" superior layout behaviors while preventing the degradation of general image quality.

Evaluation and Results

The framework was tested using Comp90, a new benchmark featuring 6,200 prompts covering 90 object categories and 12 diverse backgrounds. The results were substantial:

  • Numeracy (Counting): Accuracy increased by 29.3% for Stable Diffusion 2.1 and 19.5% for PixArt-α.
  • Spatial Relations: Stable Diffusion 2.1 saw a massive 60.7% relative increase in accuracy.
  • Human Preference: Human evaluations confirmed that the fine-tuned models outperformed baselines and rivaled complex layout-to-image methods that require manual bounding box inputs.

Conclusion and Future Directions

This research shifts the focus from prompt engineering to "seed engineering." By recognizing that random noise acts as a structural foundation, the authors provide a scalable way to improve model "common sense." While there is a potential trade-off between strict accuracy and visual diversity, fine-tuning offers a robust middle ground. Future work may extend this approach to more complex attribute binding or higher-order counting tasks.

Summary updated

BibTeX
@inproceedings{li2025enhancing,
  author = {Shuangqi Li and Hieu Le and Jingyi Xu and Mathieu Salzmann},
  title = {{Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=5BSlakturs},
}
Figure