Holding the Pixels Fixed: Does Label-Side Diversity Explain Multi-Task Transfer in Generative Recognizers?

Some image recognizers no longer pick an answer from a fixed list of classes. Instead they write the answer out word by word, the way a language model writes text. Because these generative labelers can name anything, people train them on many tasks and datasets at once: object names, region descriptions, captions, question answering. The usual justification is that "more diverse" training makes the model generalize better to categories it has never seen. Nobody has checked which kind of diversity is doing the work. The image collections differ, the label vocabularies differ, and the annotation styles differ, all at once. This proposal separates them. It trains one generative region labeler on many combinations of annotation sources that all describe the same images. The images never change, so any change in transfer to new vocabularies must come from the labels. We then ask whether a diversity score computed on the training labels alone, before any model is trained, predicts that transfer.

The practical stakes are data curation. If label-side diversity predicts open-vocabulary transfer, and does so better than simple counts like "number of tasks" or "number of unique nouns," then practitioners can choose training mixtures by measuring text instead of training dozens of models. If it does not, and the benefit reduces to how much of the test vocabulary the training labels happen to cover, that is a useful negative result. It would mean many claims about "diversity helping" are really claims about vocabulary overlap.

Core Claim

In a generative region labeler with task-conditioning tokens, the gain from adding an annotation source is predicted by how much that source increases the geometric diversity of the training target strings. The gain is measured as zero-shot performance on held-out vocabularies. We test whether the diversity score predicts beyond three simpler quantities: the number of tasks, the number of unique label words, and direct overlap with the test vocabulary. A second, subordinate claim is that the annotation-style conflict GRiT reports between COCO and Visual Genome is a label-style effect, not an image-distribution effect. If so, the conflict should appear even when the pixels are identical, and task tokens should largely remove it.

Why this, and not the obvious idea

The obvious intersection is already occupied. Answer-Me trains a T5-Large model on a mixture of vision-language tasks, all posed as text generation. It reports that more diverse task mixtures improve both known and unseen tasks, for example 35.0% zero-shot VQA2.0 accuracy from an 8-task mixture versus 25.3% for the pre-trained model. However, "diversity" there means "more datasets." Images, label vocabulary, and task format change together, so the result cannot say which ingredient mattered.

Position: Measure Dataset Diversity, Don't Just Claim It audited 135 datasets and found that 48.5% of papers infer diversity from downstream accuracy. That is exactly the circularity Answer-Me's claim inherits. The audit motivates this proposal: we compute diversity on the data before training, then test whether it predicts accuracy. This is the validity check the audit asks for.

PLDiv supplies one candidate measurement. It is a diversity score built from topological data analysis, the study of the shape of a point cloud: it sums the squared lifetimes of clusters and loops that appear and disappear as one looks at the data at coarser scales. It was validated on synthetic shapes, human-rated text diversity, and Colored MNIST, but never on vision-language labels or training mixtures. We treat it as one competing predictor, not as the answer.

The constraint that shapes the design comes from GRiT. It found that training jointly on COCO's short category names and Visual Genome's descriptive phrases hurt performance, and attributed this to conflicting "annotation philosophies." OSM sidesteps the same problem with per-dataset "mode queries." Neither work can tell whether the friction comes from the images or the labels, because each dataset brings its own images. Our fixed-image design can.

Research Program

The experimental lever is that COCO images carry many independent annotation layers. These are COCO's box annotations with 80 category names, COCO image captions, VQA2.0 question-answer pairs, and, for the substantial portion of Visual Genome drawn from COCO images, Visual Genome object names, attributes, relationships, and free-form region descriptions. Restricting to the shared images gives five or six label sources over one pixel set.

We train a GRiT-style model on subsets of these sources. It has a ViT backbone, a class-agnostic proposal stage, and a small text decoder that writes the label for each region. Each source gets its own task token, analogous to GRiT's [ObjectDet] and [DenseCap]. Image-level sources such as captions and VQA use a whole-image region. Every run uses the same number of gradient steps and the same number of images seen, so mixtures differ only in what the labels say. With six sources, all non-empty subsets containing the detection source give 32 mixtures. At a ViT-B scale this is feasible, and it provides enough points to fit and test a predictor honestly.

Before any training, we compute several scores on each mixture's target strings, embedded with a fixed sentence encoder:

  • PLDiv, averaged over bootstrap subsamples of a few thousand strings, because persistent homology scales cubically with point count.
  • The Vendi Score, a dispersion-style diversity measure, which PLDiv was reported to beat on text.
  • Unique noun count.
  • Number of tasks.
  • Coverage of each evaluation vocabulary: the fraction of test category names whose nearest training string lies within a fixed embedding distance.

The analysis is a pre-registered regression across mixtures,

$$\Delta_m = a + b\,D_m + c\,C_m + e\,K_m + \epsilon_m,$$

where $\Delta_m$ is mixture $m$'s zero-shot gain over the detection-only model, $D_m$ its diversity score, $C_m$ its test-vocabulary coverage, and $K_m$ its task count. The core claim holds only if $b$ is reliably positive with $C_m$ and $K_m$ in the model. We also require that leave-one-source-out prediction of $\Delta_m$ beats the coverage-only and count-only models.

Evaluation targets vocabularies the training sources do not define. The first is OmniLabel, which mixes plain categories, free-form descriptions, and descriptions matching nothing in the image. Our generator scores each candidate description by its length-normalized likelihood for each region, with a threshold for the no-match case. The second is multi-label recognition on OpenImages, scored with BERTScore F1 as in NXTP. In-domain COCO box AP and Visual Genome dense-captioning mAP track whether transfer gains cost in-domain accuracy.

Baselines

The naive reference is a model trained only on COCO category names with the same backbone, steps, and image set. Every mixture's $\Delta_m$ is measured against it. Any mixture that fails to beat it on OmniLabel's free-form subset shows that label diversity did nothing for that source.

The strongest existing policy for choosing a mixture is Answer-Me's: use as many tasks as possible, sampled uniformly. We run it as the all-sources mixture and as the $K_m$ predictor. Our diversity score beats it only if it ranks mixtures better than task count on held-out sources. It must also identify a smaller mixture that matches or exceeds the all-sources model on OmniLabel at equal steps.

For absolute reference points, GRiT reports 60.4 AP on COCO test-dev and 15.5 mAP on Visual Genome dense captioning with a ViT-H backbone. Our smaller backbone will not reach these, so we compare within our own runs and cite these only as the ceiling. On OmniLabel, the file reports 25.8 overall AP for the best models (GLIP-L, FIBER-B), and 4.6 free-form AP for Detic. GLIP-L falls from 33.2 to 21.2 AP when negative descriptions are included. Those are discriminative grounding models trained on far more data. A generative labeler at our scale approaching Detic's free-form number would be informative, but it is not the success criterion. No reported result was found on Lacuna for a generative region labeler on OmniLabel.

Three ablations isolate the key ingredients:

  • Task tokens removed. This tests whether style conflict appears on fixed images, which is the GRiT replication.
  • Image-varying control. We repeat four representative mixtures with each source on its own native images. This tests whether diversity effects change when images also vary.
  • Gradient-magnitude balancing added. We add the magnitude-balancing half of RotoGrad to the all-sources mixture. This checks whether a better optimizer, rather than better data selection, closes the gap.

All models share the same tuning budget: one learning-rate sweep on the detection-only model, reused everywhere.

Why not just…?

Why not just count unique label words, or measure overlap with the test vocabulary? This is the most serious threat. It may well explain most of the variance, since a model can only spell names it has been taught to spell. That is why coverage $C_m$ and noun count sit inside the regression rather than being argued away. If $b$ vanishes once $C_m$ is included, we will report that "diversity" in this setting is vocabulary coverage, which is itself a direct answer to the measurement-validity question the Position audit raises.

Why not just add more images instead of more label types? The BLOOM ablations found that a multi-source corpus beat a larger web crawl at equal token budget. That is text-only evidence, though, and NXTP's gains came from volume scaling from 3M to 70M pairs. We test this directly: a detection-only model trained on the full COCO image set, more images than the shared subset, is compared against label-diverse mixtures on fewer images at equal steps.

Why not just use task tokens or mode queries and consider the conflict solved? GRiT already used task tokens and still saw degradation, and OSM's mode queries were shown with a frozen 7B model on external proposals. Whether conditioning suffices in a small co-trained model is exactly the no-token ablation. If tokens fully remove the conflict on fixed images, the subordinate claim is confirmed, and the conflict was never about pixels.

Why not just use a contrastive model with a list of candidate names? NXTP reports CLIP at 0.450 BERTScore F1 on COCO versus 0.758 for its generator. Contrastive matching also cannot score the free-form descriptions OmniLabel emphasizes without a candidate list supplied in advance. We include a CLIP gallery baseline on the OpenImages evaluation for completeness, but it cannot answer the curation question this proposal asks.

Why not just fix negative transfer with gradient homogenization? RotoGrad has only been tested with closed-set heads. It may help, which is why its magnitude-balancing half is an ablation. If it closes the gap between the all-sources and best-selected mixtures, then mixture selection matters less than optimization, and we say so.

Decisive Checks

The central check is whether PLDiv or Vendi on target strings predicts held-out-source zero-shot gain on OmniLabel after controlling for coverage and task count. The pre-registered threshold is a positive coefficient whose bootstrap interval excludes zero, together with lower leave-one-source-out error than the coverage-only model. The second check is whether dropping task tokens produces the GRiT-style in-domain drop on fixed images. The third is whether the diversity-selected mixture matches the all-sources mixture on OmniLabel negatives, the setting where hallucinated matches are penalized. Failing the first check while passing the second would still settle a question the field currently answers by assertion.