TL;DR
Text-to-image (T2I) models exhibit a "reach" for counting—a threshold, typically around ten objects, beyond which number words lose their ability to steer the generation process. This research demonstrates that past this reach, the coarse starting noise fixes the image layout, while the final count is left to be determined by fine-scale noise, rendering the specific number word in the prompt largely ineffective.
What Problem Does This Paper Address?
While it is well-known that diffusion-based T2I models struggle with counting, the nature of this failure is poorly understood. Does the model simply ignore the number word at high counts, or does it try and fail? Standard evaluation metrics like "exact-match" are insufficient because even a model that follows a number loosely will rarely be exact past ten objects, making it indistinguishable from a model that ignores the number entirely.
The authors investigate whether the number word still "does anything" at high counts and how the interaction between the prompt and the initial latent noise dictates the final count. They aim to move beyond simple accuracy scores to understand the internal dynamics of counting in state-of-the-art open models.
Methods
The researchers introduced a "same-noise intervention" protocol. Because image generation is deterministic given a fixed seed and starting noise, they could isolate the effect of the number word by keeping the starting noise constant while changing only the requested count.
Key techniques included:
- Same-Noise Order: A new metric measuring how often a larger requested count actually results in more objects when the starting noise is held constant.
- Fourier Noise Decomposition: They used 2D Fourier transforms to separate "coarse" noise (low spatial frequency) from "fine-scale" noise (high spatial frequency). This allowed them to test which part of the noise determines the layout versus the count.
- Planted Starts: Instead of starting from pure noise, they initialized the model with a noised grid of the target object to see if a pre-defined layout could "carry" the count information better than a text prompt.
- Human-in-the-loop Counting: Recognizing that automated object detectors (like SAM 3 or Grounding DINO) often undercount dense crowds, the authors had humans count 3,088 generated images to establish a reliable ground truth.
Datasets
The study utilized:
- LVIS Category Names: 64 everyday object types (e.g., "desk," "coil," "artichoke") were used to generate prompts.
- Human-Counted Image Set: A newly created dataset of 3,088 images with human-verified counts, which the authors plan to release.
- Models Evaluated: Stable Diffusion 3.5-Medium (SD3.5-M), SDXL, FLUX.1-dev, FLUX.2 [klein], and Qwen-Image.
Key Insights and Ideas
- The "Reach" of a Model: Every model follows numbers accurately up to a point. For SD3.5-M and FLUX.1-dev, this "reach" ends shortly after ten. FLUX.2 and Qwen-Image follow requests loosely up to twenty-five.
- Layout vs. Count Split: In models like SD3.5-M, the coarse part of the starting noise fixes the composition (the layout), but not the count. Past the reach, changing the number word moves the count about as much as changing the random seed, but without a consistent direction.
- Fine-Scale Noise Dominance: Redrawing only the fine-scale noise (finer than 1/8th of the image) keeps the layout intact but moves the count two-thirds as far as a completely new number word would. This suggests the count is "decided" by high-frequency details in the noise.
- "Showing" Beats "Telling": Models are far more successful when the count is provided as a layout in the starting latent space rather than a word in the prompt. A "planted" grid of objects resulted in 59% exact counts, compared to only 4% when the count was only specified in text.
Key Results
- Order Metrics: Within the reach (2–8 objects), models showed high "same-noise order" (80–99%), meaning higher requests consistently led to more objects. Past the reach, this dropped significantly (e.g., 0.57 for SD3.5-M, where 0.50 is random).
- Reach Comparison: Figure 3 illustrates how different models diverge past ten. While SD3.5-M and FLUX.1-dev plateau or even show a decrease in median count for high requests, Qwen-Image and FLUX.2 continue to follow the request, albeit loosely.
- Early Action, No Direction: Timing tests showed that number words act in the first few denoising steps. However, for large numbers, this action lacks "direction"—it changes the image but doesn't reliably push the count toward the requested value.
Limitations and Future Work
- Scope: The study focused on single-object prompts and a limited set of five open models. Closed models (like Midjourney or DALL-E 3) were not tested.
- Prompt Engineering: Only one prompt template was used; it's possible different phrasing could extend a model's reach.
- Human Noise: Human counting itself becomes noisy and inconsistent for very dense crowds (25+ objects), which can resolve some model rankings but leave others ambiguous.
- Future Directions: The authors suggest that future benchmarks for counting should use "same-noise" tests rather than simple accuracy. They also point toward improving layout-based conditioning as a more promising path for high-count generation than text-only prompting.
Summary updated