Ranking Coordinate-Token Vision Models Before Training Them

Some vision systems treat object detection and visual grounding as text generation. Pix2seq-style models cut the image plane into a grid of numbered bins, so each box corner becomes a word like <loc_384>. A plain Transformer then learns to write out boxes token by token, trained only with the usual next-word loss. These models are simple and competitive. Their architecture, however, is chosen by hand, and it has knobs that ordinary vision or language models lack: how many bins to use, and how to split compute between the image encoder and the token-writing decoder. These choices set both accuracy and how slow the model is to decode. Picking them by fully training many candidates is expensive, and cheap shortcuts from architecture search have never been checked on this kind of model.

This proposal argues that the standard cheap signals break here for a specific reason. The training loss, cross-entropy over location tokens, cannot be compared across different bin counts. It also treats a near-miss bin the same as a wild miss. Parameter count, meanwhile, cannot tell encoder capacity from decoder capacity. We propose a geometry-aware early-training score: the expected coordinate error under the model's predicted bin distribution, measured after a few hundred training steps. We claim this score ranks candidate architectures by their final localization accuracy far better than parameter count, zero-cost scores, or early validation loss at the same compute. To test the claim, we build the first small tabular benchmark of fully trained coordinate-token models.

Background and the gap

The tokenization side supplies the model family. Pix2seq writes each object as five tokens (four quantized coordinates plus a class). It reaches 43.0 AP on COCO with a ResNet-50 trained from scratch. A Unified Sequence Interface for Vision Tasks extends the same interface to masks, keypoints and captions, reaching 46.5 detection AP with a 132M-parameter model. SeqTR applies it to referring-expression grounding, reaching 71.70 mIoU on RefCOCO validation for segmentation. All three fix the bin count and the encoder/decoder split by hand. SeqTR, for example, uses 1,000 bins with 6 encoder and 3 decoder layers. Pix2seq and the unified-interface paper both name slow token-by-token decoding as their main weakness. LayoutNUWA shows that bins need not be uniform, since k-means bins work for layouts. That makes the quantization scheme a real design axis rather than a constant. These papers motivate the work: they expose a design space nobody has searched.

The search side supplies the cheap evaluators and also the warnings. Zero-Cost Proxies for Lightweight NAS scores untrained networks from a single minibatch. A vote over three such scores reaches Spearman 0.82 against final accuracy on NAS-Bench-201. NAS-Bench-Suite-Zero standardizes 13 such scores and shows two things: they lean heavily toward bigger models, and feeding them into a learned predictor improves rank correlation by up to 42%. LiteTransformerSearch finds that, for decoder-only language models, parameter count alone ranks perplexity with Spearman 0.97 to 0.99. It also finds that latency differs sharply among equal-size models, and it names encoder-decoder and multimodal settings as open. TransNAS-Bench-101 shows that architecture rankings learned on classification can transfer worse than random to geometric tasks such as room-layout regression. That result constrains us: we should not assume any classification-validated proxy carries over.

The closest existing systems each stop short of this question. ViTAS searches Vision Transformer backbones with weight sharing but pairs them with conventional dense detection heads. FBNetV2 searches input resolution, but over continuous feature maps rather than output vocabularies. Neither touches the decoder, the bin count, or sequence outputs.

Core Claim

When candidate architectures differ in vocabulary, the training loss stops being a valid ranking signal. With $B$ bins over a unit interval, a model that places all its mass on the right bin but spreads it uniformly within that bin still pays a cross-entropy cost that grows like $\log B$. Finer bins therefore look worse even when the geometry is identical. A cheap ranking signal for these models must be measured in image space, not token space. We define the proxy, computed under teacher forcing on held-out images after a short training budget $T$, as

$$ G_T = \mathbb{E}_{(x, y)} \Big[ \sum_{k=1}^{B} p_k(x) \, \lvert c_k - y \rvert \Big]. $$

Here $y$ is a ground-truth normalized coordinate, $c_k$ is the center of bin $k$, and $p_k(x)$ is the model's predicted probability for bin $k$ given the image and preceding tokens. This is the expected absolute coordinate error, and it is the one-dimensional earth-mover distance from the predicted distribution to the truth. It has three properties the alternatives lack. It is comparable across bin counts and across uniform or k-means bins. It rewards probability mass placed near the truth. It gives a dense per-token signal long before full-sequence decoding produces any nonzero AP.

The claim is that $G_T$ predicts final detection AP and grounding accuracy across this search space. Raw cross-entropy and parameter count should fail at it, and so should each of the following on its own: CE corrected for bin width, zero-cost scores, and early AP.

Research Program

The first deliverable is a small tabular benchmark, needed because no ground truth exists. The search space has four parts. The image encoder varies in depth and width, in a ViT-small-to-base range. The decoder varies in depth, width and head count, independently of the encoder. The bin count is drawn from {100, 250, 500, 1000}. The quantization is either uniform or k-means fitted on training coordinates, following LayoutNUWA's approach.

We train roughly 300 sampled configurations to convergence under a reduced protocol: 384-pixel inputs, a fixed schedule, and a single seed, with three seeds for a 30-model subset to estimate noise. The two tasks are referring-expression comprehension on RefCOCO/RefCOCO+/RefCOCOg, in the SeqTR four-token box format, and COCO 2017 detection, in the Pix2seq five-token format. We log five things for every model: final accuracy or AP, wall-clock decoding latency measured on a fixed GPU at batch size one as LiteTransformerSearch does, peak memory, and checkpoints at several early budgets $T$.

Reduced-protocol ground truth is itself a low-fidelity estimate. We therefore retrain the top and bottom ten configurations under the full Pix2seq schedule to check that the rankings survive.

The second deliverable is the proxy study. At matched proxy cost, we compute $G_T$ and all competitors and report Spearman and Kendall correlation with final AP. We also report top-10% recall, which matters more for search than global correlation. We do this separately within fixed bin counts and across bin counts, because the comparison across bin counts is where we predict raw cross-entropy collapses.

The third deliverable is a latency-aware search. We use evolutionary search ranked by $G_T$ and constrained by measured decoding latency, find the accuracy-latency Pareto front, and fully train the chosen models. This tests the practical question the tokenization papers raise: can the encoder/decoder split and bin count be traded to make decoding faster without losing accuracy?

Baselines

The do-nothing reference is the hand-designed configuration: a Pix2seq-style encoder with a standard 6-layer decoder and 1,000 uniform bins, retrained under our exact reduced protocol. Published figures provide the full-protocol anchor. For COCO detection, Pix2seq reports 43.0 AP with ResNet-50 from scratch, and the unified interface reports 46.5 AP with Objects365 pretraining. For grounding, SeqTR reports a +6.27 absolute gain over TransVG on RefCOCOg comprehension, and the file gives no absolute REC accuracy for SeqTR. A searched model beats this reference if it reaches equal AP at clearly lower measured latency, or higher AP at equal latency, under the same training budget.

The strongest scale-based competitor is parameter count. FLOPs count is included as well, since NAS-Bench-Suite-Zero finds it among the most robust single proxies. Its strength comes from LiteTransformerSearch, which reports 0.97–0.99 Spearman on language modeling perplexity. For coordinate-token models, the file has no reported result on Lacuna for parameter-count ranking. We report total, encoder-only and decoder-only counts.

The strongest learned competitor is the NAS-Bench-Suite-Zero recipe: all standard zero-cost scores plus FLOPs fed into an XGBoost predictor, trained on the same number of labeled benchmark architectures that any learned variant of our method would use. Its best published single-benchmark figure is the 0.82 Spearman vote reported by Zero-Cost Proxies for Lightweight NAS on NAS-Bench-201. That number comes from classification, and the file has no reported result on Lacuna for zero-cost proxies on spatial-token models.

The low-fidelity competitors are early validation cross-entropy and early AP, both at the same $T$ and on the same held-out images. Two ablations isolate the key ingredient. The first is bin-width-corrected cross-entropy, CE plus the log of the bin width. This approximates continuous negative log-likelihood and fixes cross-vocabulary comparability but gives no credit for near misses. If $G_T$ beats it, the gain comes from the geometric distance itself and not merely from normalization. The second applies $G_T$ to a single encoder/decoder split, removing the bin axis. This shows how much of the advantage appears only when vocabularies differ.

$G_T$ counts as winning if it beats every competitor's top-10% recall and Spearman correlation across bin counts at matched compute, with the margin exceeding the seed-noise band estimated from the three-seed subset.

Why not just…?

Why not just use an off-the-shelf ViT and a standard decoder, as Pix2seq intends? That is our do-nothing baseline, and it may be close to optimal on accuracy. The case against it concerns latency. LiteTransformerSearch found 1.5–2.5× runtime gains from non-uniform layer shapes at equal quality, and Pix2seq names decoding speed as its main cost. If the Pareto front from our search does not dominate the hand design on measured latency, we will report that the default is good enough.

Why not just rank by parameter count? This is the most serious threat within a fixed vocabulary, where it may well match $G_T$. We expect it to fail across vocabularies, because bin count barely changes parameters but strongly changes precision and latency. It also cannot express where capacity sits. The within-bin and across-bin correlation split settles this directly. If parameter count wins within fixed bins, we will say so and restrict our claim to the vocabulary and split axes.

Why not just train each candidate briefly and measure AP? This is the honest alternative to our proxy. At very short budgets, full-sequence AP requires every coordinate and the class to be right at once, so we expect it to sit near zero and be noisy. $G_T$ uses every token's full distribution. Early AP may catch up at larger $T$. We therefore plot rank correlation against $T$ for both and report the crossover budget, if one exists. That curve, not a single budget, is the result.

Why not just use a weight-sharing supernet? A supernet would need to share weights across output vocabularies of different sizes, which no existing method does. Transformer supernets already need special fixes such as ViTAS's cyclic sharing. Differentiable search also has documented collapse modes, shown by NAS-Bench-201. We leave supernets as future work once the benchmark exists to evaluate them.

Proposed Experiments

The decisive check is the correlation of each proxy with final reduced-protocol AP, computed across all bin counts on COCO and RefCOCO at matched compute. We predict raw cross-entropy will rank poorly once bin counts are mixed. We predict corrected cross-entropy will recover part of the gap, and $G_T$ will recover most of it. If corrected cross-entropy matches $G_T$, the geometric-distance idea is unnecessary and normalization is the whole story.

The second check is transfer. Rankings built on RefCOCO grounding are tested on COCO detection, mirroring the TransNAS-Bench-101 question but for token outputs. The third check is whether reduced-protocol rankings survive full-schedule retraining for the extreme configurations. The final check is the latency-constrained search itself. Its chosen models are fully trained and placed against the hand-designed baseline on measured accuracy and latency, using the full-protocol training recipe for both.