Executive Summary
Researchers have discovered that the "random" noise used to initialize image generation in diffusion models isn't entirely neutral. Instead, certain random seeds act as "invisible templates" that naturally favor specific object counts and spatial layouts. By mining these reliable seeds and fine-tuning models on their own most successful outputs, the authors achieved massive improvements in compositional accuracy—including up to a 60% boost in spatial relations—without requiring any manual human annotation.
The Compositional Generation Problem
Current text-to-image (T2I) diffusion models, such as Stable Diffusion, excel at artistic generation but often struggle with basic counting and spatial reasoning. If prompted for "four unicorns," a model might generate three or five. If asked for a "penguin on the right of a bowl," it frequently flips the positions or merges the concepts haphazardly.
This challenge is known as the compositional generation problem. While previous research has attempted to fix this by forcing models to follow strict layouts or using Large Language Models (LLMs) to plan bounding boxes, these methods are often slow, labor-intensive, or restrict the model's creative flexibility. This paper asks a fundamental question: Can we improve a model's compositional ability simply by understanding and leveraging the initial noise it starts with?
Methodology: Mining and Fine-Tuning
The authors introduce a three-step framework to turn "lucky" noise into a permanent skill for the model:
- Reliable Seed Mining: The team generated thousands of images using different random seeds. They then employed a Vision-Language Model (VLM), specifically CogVLM2, to act as an automated judge. The VLM verified whether the generated image actually contained the requested number of objects or the correct spatial relationship. Seeds that consistently produced correct results across various prompts were labeled "reliable."
- Inference-time Sampling: The most straightforward application of this discovery is to use these "golden seeds" during generation. For instance, if a user wants four objects, the system can initialize the generation with a seed that the mining process identified as favoring a "four-grid" layout.
- Fine-Tuning on Self-Generated Data: To improve the model's baseline performance regardless of the seed used, the researchers fine-tuned it on a curated dataset of its own successful "reliable" images. Crucially, they only updated the Query (Q) and Key (K) projection layers within the attention modules. This surgical fine-tuning preserves the model's overall image quality while specifically sharpening its understanding of how text relates to object placement.
The Comp90 Benchmark
To rigorously test their method, the authors introduced Comp90, a new benchmark specifically designed for compositional evaluation:
- Categories: 90 distinct object types (animals, food, common objects).
- Backgrounds: 12 hand-crafted settings (e.g., "on a rocky alien planet").
- Prompts: 3,000 prompts for numerical composition (counting 2–6) and 3,200 for spatial relations (top, under, left, right).
- Split: 60 categories were used for training/mining, and 30 "unseen" categories were reserved for testing to ensure the model learned the concept of counting and placement, rather than just memorizing how to draw specific training objects.
Key Insights: Seeds as Layouts
The most striking finding of this research is that initial noise patterns are not just random static; they actively guide the model toward specific "camera angles" and compositions.
As seen above, a specific seed (like seed = 50) consistently produces a two-object layout slightly offset from the center, whether the prompt asks for apples, oranges, coins, or umbrellas. Another seed (seed = 23) consistently scatters three objects.
Furthermore, the authors identified a "Four-Grid" Phenomenon. Certain seeds naturally partition the image space into quadrants. These seeds are significantly more reliable for prompts requiring exactly four objects because they provide a natural "slot" for each item. By using a VLM to "rectify" prompts (e.g., if a model draws five apples for a "four apples" prompt, the VLM re-labels that image as "five apples" for training), the model learns from its mistakes rather than being penalized for them.
Results and Performance
The proposed method was tested on Stable Diffusion 2.1 and PixArt-α, demonstrating dramatic improvements:
- Numerical Accuracy: Stable Diffusion achieved a 29.3% relative increase in counting accuracy, while PixArt-α saw a 19.5% increase.
- Spatial Accuracy: The gains in spatial composition were even more substantial, with a 60.7% relative increase for Stable Diffusion (moving from 17.8% to 36.6% accuracy) and a 21.1% increase for PixArt-α.
- Generalization: The fine-tuned models improved even on "out-of-scope" quantities (like asking for 7 or 8 objects) and multiple-category prompts (e.g., "three tigers and two penguins"), proving they learned general compositional rules.
- Aesthetics: Unlike other layout-control methods that can produce robotic or "glitchy" images, this self-supervised fine-tuning method maintained, and sometimes even improved, the aesthetic quality of the generated images.
Limitations and Future Work
While highly effective, the approach has a few limitations. First, there is a trade-off between diversity and accuracy: relying exclusively on a small set of "reliable" seeds can lead to repetitive layouts. The authors mitigate this through fine-tuning, but the tension between strict prompt adherence and creative variety remains. Second, the system's success is bottlenecked by the VLM's accuracy; if the VLM struggles to count or understand complex spatial relations, those errors will be baked into the fine-tuned T2I model. Finally, while the model handles simple "A on top of B" relationships well, highly complex scenes with numerous interacting objects remain a challenge for future iterations of seed mining.
Summary updated