Clicks Where the Generator Fails: Budgeted Point Supervision for Compositional Diffusion Transformers

Text-to-image generators have become very good at drawing a single object and remain poor at putting several objects in the right places with the right colors. GenEval, an object-detector-based benchmark, finds that larger models approach ~98% on single-object prompts but sit near ~15% on spatial position and ~35% on attribute binding, with little gain from scale (GenEval). The most direct fix so far is to supervise the model's internal attention, the per-word maps showing which image region each word influences, with object masks. TokenCompose does this using masks generated automatically by a large segmentation system (TokenCompose). This ties the generator to the segmenter's mistakes. It also treats every training image as equally worth labeling.

We propose to spend a small human budget of single clicks, roughly one per mentioned object, and only on the training images where the generator's own attention shows it is confused about which word owns which region. Each click is turned into a dense attention target inside the generator, without an external teacher. A global assignment step, optimal transport, splits image regions among the clicked objects so that two cups in one picture are kept apart. The thing we measure is not a single score. It is a cost curve: how much compositional accuracy a diffusion transformer gains per hour of human annotation, compared with free automatic masks, expensive dense masks, and randomly placed clicks.

Core Claim

Grounding fine-tuning of a diffusion transformer has two properties that make sparse, targeted supervision viable. A diffusion transformer is a denoising image generator built from standard transformer blocks over image patches (Scalable Diffusion Models with Transformers).

First, the error it needs to correct is mostly about ownership: which word claims which tokens, especially when several instances or attributes compete. It is much less about exact boundaries. A click per instance carries the ownership information. Dense masks add boundary detail that we expect matters little for GenEval-style correctness.

Second, the model's own uncertainty identifies the failures. Comparing attention maps across noise draws and timesteps flags the images where words compete for the same region. This lets an annotator label a small fraction of the data and still recover most of the gain.

The new element is the combination: active, model-driven selection of what to click, plus click-to-target propagation computed from the generator's own features. Prior work does neither for generative training. Two concrete predictions follow, and both can fail. At equal annotation hours, model-selected clicks should beat both random clicks and dense masks. On prompts with multiple same-class instances carrying different attributes, clicks should beat fully automatic segmenter masks even though those masks cost no human time.

Research Program

Base model and data. The base generator is a public text-to-image diffusion transformer. Stable Diffusion v3 Medium is the natural choice because DiT-ST reports its GenEval score (Split-Text Conditioning). In its joint attention we read the image-token-to-text-token block as the per-word maps.

Training images and captions come from COCO (Microsoft COCO). COCO matters here because its human instance masks let us simulate annotators exactly. A simulated click is a point sampled inside the instance mask, linked to the caption word naming it. The same masks give us a dense-supervision oracle at known human cost. We price annotation using Point2Mask's measured costs of ~22 seconds per image for points versus 240 seconds for full masks (Point2Mask).

Selection step. For each candidate image-caption pair, we noise the image latent at several timesteps with several noise seeds and read the attention map of every object word. The score combines two signals: how much each word's map varies across seeds, and how much the maps of different object words overlap. The second signal targets binding conflicts directly.

This adapts the logic of LabOR, which queries labels only where two auxiliary classifiers disagree and reaches 63.5% mIoU on GTA5-to-Cityscapes with ~40 points per image (LabOR). We replace classifier disagreement with disagreement across the generator's own noise draws. Unlike LabOR, we run selection offline in rounds of a fixed size, so no annotator has to be on call during training.

Propagation step. Given the clicks $p_1,\dots,p_K$ for words $w_1,\dots,w_K$, we solve an entropic optimal transport problem with Sinkhorn iterations, following Point2Mask's supplier–consumer formulation. Each click, plus one background supplier, offers labels, and each image token consumes one. The cost combines the distance between a token's features and a clicked token's features with their spatial distance.

The difference from Point2Mask is where the features come from. Point2Mask builds costs from a segmentation network's maps. We take features from an exponential-moving-average copy of the generator itself, with gradients stopped. This follows Self-Transcendence's evidence that a diffusion transformer's own intermediate features are good enough to supervise its training (Beyond External Guidance).

The resulting soft assignment $T_k$ is the share of each token given to click $k$. It becomes the target for word $w_k$'s normalized attention map $A_k$:

$$\mathcal{L} = \mathcal{L}_{\text{denoise}} + \alpha \sum_{k=1}^{K} \mathrm{KL}\left(T_k \,\|\, A_k\right) - \beta \sum_{k=1}^{K} \log A_k(p_k).$$

Here $\mathcal{L}_{\text{denoise}}$ is the standard diffusion loss. The KL term pulls each attention map toward its assigned region. The last term guarantees that each word's attention covers its own clicked token, in the spirit of the point-plus-objectness losses of Point-supervised Segmentation via Objectness Regularization. The weights $\alpha$ and $\beta$ are tuned on a held-out split.

Scope. The study varies the click budget, the selection rule, and the propagation rule. It reports compositional accuracy, image quality, and training overhead as functions of annotation hours.

Baselines

The naive reference point is the base diffusion transformer fine-tuned on the same COCO image-caption pairs for the same number of steps with only the denoising loss. This separates the effect of grounding supervision from the effect of extra in-domain training.

Automatic dense masks (strongest competitor on cost). TokenCompose ported to the diffusion transformer: Grounded SAM pseudo-masks for every caption noun, with its token-level and pixel-level attention losses, on the same images. This is the strongest competitor because it costs zero human time and already nearly doubles VISOR on a U-Net (52.15% vs 29.86% for base Stable Diffusion, reported in TokenCompose). There is no reported result found on Lacuna for TokenCompose on a diffusion-transformer backbone, so we produce this number ourselves. It gets the same steps, learning-rate search, and loss-weight search as our method. Beating it means higher GenEval position and attribute-binding accuracy at matched FID, or equal accuracy with fewer training images.

Strongest compositional result in the file. DiT-ST reports 69% GenEval overall versus 62% for Stable Diffusion v3 Medium (Split-Text Conditioning). DiT-ST changes how text is scheduled rather than where words land, so it is complementary. We compare against the reported 62% base and 69% DiT-ST figures, and where reproducible, also stack our loss on top of split-text conditioning.

Dense human masks (oracle). The same attention losses trained on COCO's true instance masks. This is the quality ceiling and the expensive end of the cost curve.

Inference-time control. CAC, which masks attention at sampling time using user boxes (Cross Attention Control). It is given ground-truth boxes on the evaluation set and its λ is tuned. This tests whether any training is needed.

Ablations. The ablations isolate each key ingredient. Random clicks at the same budget test the selection rule. Voronoi expansion of clicks and point-only loss without propagation test the transport step. Grounded SAM features in place of the generator's own features test teacher-free propagation.

Why not just…?

Why not just run Grounded SAM on everything, as TokenCompose does? This is the main threat. If automatic masks match human clicks on every evaluation slice, human annotation is not worth buying, and we will report that. The case for clicks rests on failure modes. A noun-level open-vocabulary detector has no way to say which of two cups is "the red cup," and TokenCompose supervises nouns only, so instance-level attribute binding is outside what it encodes. Its accuracy is also capped by the teacher's detection errors. We settle the question with a stratified evaluation that reports results separately on prompts with repeated categories and differing attributes, and on images where Grounded SAM misses an annotated COCO instance. A hybrid arm, using automatic masks everywhere plus clicks only on the actively selected images, tests whether clicks are best used as corrections.

Why not just control attention at inference time? CAC raises CC-500 binding from 49.91% to 67.27% at under 0.1 s of extra cost (Cross Attention Control). However, it needs a box from the user for every generation, degrades on small or overlapping regions, and leaves the model unchanged. Our comparison gives CAC oracle boxes and asks whether a trained model with no boxes at test time matches it on GenEval position. A trained model that reaches parity with plain prompts would be the more useful system. If CAC with oracle boxes is far ahead, that bounds what training can buy.

Why not just scale the model or the data? GenEval reports that larger models leave spatial position near 15% (GenEval). The naive same-data fine-tuning baseline also checks whether extra in-domain steps alone move the scores.

Why not just click randomly or densify clicks with simple geometry? If random clicks match model-selected clicks, the selection contribution disappears, and the random-click ablation tests this directly. Voronoi-style expansion is known to fail between adjacent same-class objects in segmentation (Point2Mask; Objectness Regularization). We verify whether that failure carries over to attention targets.

Proposed Experiments

We train every arm at click budgets corresponding to roughly 1%, 5%, and 20% of COCO training images, plus the oracle at 100%. Each arm's annotation cost is converted to hours using the 22 s and 240 s figures. Evaluation uses GenEval (overall plus the position, counting, color, and attribute-binding subtasks), VISOR, MultiGen, and CC-500 for composition. Image quality uses FID and CLIPScore on COCO-5K, and we also report training-time overhead from the Sinkhorn step. Each result is averaged over three seeds with confidence intervals.

The decisive checks are three. First, at equal annotation hours, actively selected clicks should beat random clicks and dense masks on GenEval position and attribute binding. If they do not, the budgeting thesis fails. Second, on the repeated-category, differing-attribute slice, clicks with self-feature transport should beat the Grounded SAM baseline at matched FID. If they do not, automatic masks suffice and the cost case collapses. Third, replacing the generator's own features with teacher features in the transport cost should not help. If it does, the teacher-free claim is wrong, and we would recommend the cheaper hybrid instead.