Prompt a state-of-the-art diffusion model to draw "three apples on a plate," and you will frequently receive an image containing two, four, or a cluster of ambiguous shapes. This breakdown in quantitative precision—often called the numerosity problem—is a persistent blind spot across massive foundation models (Yuan et al.). While modern generative models excel at rendering complex textures and high-level semantics, they fundamentally struggle to map discrete cardinal numbers to visual entities (Jiang et al., Haydarov et al.).
This failure spans text-to-image generation (Holynski et al.), image editing (Tan et al.), and 3D scene synthesis (Cui et al.). It even introduces severe label noise when researchers attempt to use diffusion models to generate synthetic training data for downstream counting tasks (D'Alessandro et al., Zheng et al.).
To solve this problem, we must first understand the underlying mechanics of why diffusion models are inherently bad at counting.
The Mechanics of the Counting Blind Spot
The inability to count is not a trivial bug that can be fixed by simply scaling up models or datasets (Yuan et al.). Instead, it arises from a fundamental mismatch between how text encoders represent numbers and how diffusion processes generate pixels.
1. Semantic Blending in Text Encoders
Standard text encoders (such as CLIP) process text as continuous semantic vectors rather than discrete logical tokens. In this embedding space, the difference between "three cats" and "four cats" is often washed out. The high-level semantic concept of "cat-ness" dominates the embedding, overriding the strict mathematical logic of the cardinal number (Chen 0001 et al.).
As Keuper et al. note, the semantic representation of quantity in the embedding space is insufficiently distinct for the diffusion process to resolve accurately. Because the text encoder fails to distinguish between individual entities and groups, the downstream model loses track of object cardinality (Long et al.).
2. The Subitizing Limit
Generative models exhibit a sharp decline in accuracy as the requested count increases (Yuan et al.). This behavior closely mirrors the human psychological phenomenon of subitizing—our ability to instantly perceive small quantities (usually 1 to 4) without explicitly counting them.
When prompted for small numbers, diffusion models can occasionally succeed by relying on learned visual patterns of small groups. However, for higher counts, they default to a coarse perception of "few" versus "many," lacking the discrete logic required for exact high-count generation.
3. The Tyranny of Initial Noise
A deeper architectural issue lies within the denoising process itself. Diffusion models begin with a canvas of pure Gaussian noise ($z_T$) and iteratively shape it into an image. Yuan et al. demonstrated that this random noise initialization exerts a dominant, biased influence on object counts that frequently overrides textual instructions.
If the initial random noise layout naturally contains three high-frequency energy clusters that look like proto-objects, the model will struggle to force a fourth object into existence, regardless of what the prompt demands. The text prompt is simply not strong enough to reshape the geometric layout pre-determined by the random seed.
4. Continuous Spaces vs. Discrete Boundaries
Diffusion models operate by shifting continuous probability distributions. They lack explicit mechanisms to enforce discrete logical boundaries within these continuous latent spaces (Gupta et al.). During the iterative denoising process, distinct objects often merge together or duplicate because the model lacks a mechanism to track object identity over time (Tan et al.).
Mechanisms for Precise Numerical Grounding
To overcome these limitations, recent research focuses on injecting discrete spatial and logical constraints into the continuous diffusion process. These approaches broadly fall into three categories.
Explicit Layout Planning
If text encoders cannot represent numbers precisely, the task of spatial planning can be offloaded to a model that can. Frameworks like LayoutLLM-T2I and InstanceGen use Large Language Models (LLMs) as "architects."
[Text Prompt: "Three apples"] ──> [LLM Planner] ──> [Explicit Bounding Boxes] ──> [Diffusion Denoising] ──> [Accurate Image]
The LLM parses the prompt and generates explicit bounding boxes or instance-level layouts before the diffusion model begins painting pixels. This decouples quantitative reasoning from pixel synthesis, ensuring that each requested object has a dedicated spatial slot.
Attention Division and Binding
Other approaches intervene directly during inference by manipulating cross-attention maps, which dictate which parts of the image respond to specific words in the prompt.
The "Divide & Bind" technique introduced by Keuper et al. forces cross-attention maps to allocate separate, non-overlapping spatial regions for distinct objects, preventing them from blending together. Similarly, the Quantity Attention (QTTN) module designed by Tan et al. dynamically tracks and maintains object counts during multi-object image editing, preventing the unintended merging or duplication of subjects.
Topological and Reinforcement Constraints
To prevent objects from disappearing into the background, researchers are introducing strict structural and behavioral boundaries:
- Topological Constraints: Gupta et al. integrated topological constraints via persistent homology directly into the diffusion process. This mathematically guarantees that the generated image adheres to a specific number of distinct connected components or enclosed regions, forcing the continuous latent space to respect discrete counts.
- Reinforcement Learning: Borse et al. utilized reinforcement learning with diversity constraints (via Group-Relative Policy Optimization). By penalizing the model when it duplicates identities or fails to meet the exact person count in multi-human scenes, they train the model to respect discrete boundaries.
By shifting from purely semantic text-to-image mapping to structured, layout-driven, and topologically constrained generation, these methods bridge the gap between continuous visual synthesis and discrete numerical logic.
Key Foundational Work
LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation (2023) — contributed 1 concept elements to this direction.
VICTR: Visual Information Captured Text Representation for Text-to-Vision Multimodal Tasks (2020) — contributed 1 concept elements to this direction.
SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models (2023) — contributed 1 concept elements to this direction.