Next-scale image generators predict an image as a stack of token maps, running from a single coarse token up to a 16×16 grid of fine ones. They are trained with ordinary classification loss, which scores every wrong token as equally wrong. That is a strange choice for images. Swapping one coarse-scale token for a slightly different one nudges the colour of a whole quadrant. Swapping it for an unrelated one breaks the layout. The loss cannot tell these apart. Distance-aware losses give "partial credit" for predictions near the target: the training target is spread over tokens close to the correct one instead of placed entirely on it. They help in many structured-output problems. In image generation, however, the one direct test gave almost nothing, improving LlamaGen's FID only from 4.24 to 4.18 (Teaching Metric Distance to Discrete Autoregressive Language Models). We argue that this small gain came from the wrong distance, not from a weak idea. We also argue that the question that matters for scaling has not been asked: does partial credit shift the scaling curve by a constant amount of compute, or does it change its slope?
We propose to measure token distance by what a token swap does to the decoded image at the scale where it happens. We will then train a family of next-scale models of increasing size with and without this signal and fit scaling curves to both. The central claim is that a scale-aware image-space distance gives a compute saving that does not shrink as models grow, while scale-agnostic codebook distances and uniform smoothing give gains that fade with size. Either outcome is informative. A fading gain would tell practitioners to simply scale up. A persistent or growing gain would make the training objective a real scaling lever for discrete visual generation.
Core Claim
Visual Autoregressive Modeling (VAR) replaced raster-order token prediction with coarse-to-fine "next-scale prediction". It reported a -0.998 correlation between compute and performance and an ImageNet 256×256 FID of 1.73. It uses one codebook shared across all scales, and it trains every scale with plain cross-entropy over codebook indices.
Sharing the codebook matters here. The same pair of codes means very different things at different scales. At the 1×1 scale, a code is upsampled to cover the whole image. At the 16×16 scale, it covers one small patch. Any single static distance between codes, whether in codebook embedding space or from the cosine similarity used by DIST²Loss, therefore gives the same credit to a near-miss at every scale. That is wrong at both ends.
A second problem compounds the first. Exploiting Discriminative Codebook Prior for Autoregressive Image Generation showed that Euclidean geometry in codebook space is itself misleading, because the codebook's density is uneven and cluster centres often fall in empty regions. Our claim is that the right distance lives in the space the tokenizer actually adds up, and that it has to be computed separately for each scale.
Concretely, VAR's tokenizer upsamples each scale's token map to full resolution, passes it through a small scale-specific convolution $\phi_k$, and sums the results before decoding. Because this path is linear, the change caused by swapping code $i$ for code $j$ at one position of scale $k$ can be computed exactly and without any image context:
$$d_k(i,j) = \big\lVert \phi_k\big(\mathrm{up}_k(e_j - e_i)\big) \big\rVert_2 ,$$
Here $e_i$ is the embedding of code $i$, $\mathrm{up}_k$ is the upsampling of a single-position map from scale $k$ to the final feature resolution, and the norm is taken over the full feature map. A variant replaces the norm with a decoder-linearized pixel distance, estimated once per scale by averaging decoder Jacobians over a few thousand training images.
The training target at scale $k$ for true code $i$ is
$$q_k(j \mid i) \propto \exp\!\big(-d_k(i,j)^2 / \tau_k\big),$$
and the loss is $\mathcal{L} = \mathcal{L}_{CE} + \alpha\,\mathrm{KL}(q_k \,\|\, p_\theta)$. This is the same additive form as DIST²Loss, so the comparison with it is clean. To avoid a separate hyperparameter per scale, $\tau_k$ is set by one shared rule: $\tau_k$ equals $\beta$ times the median squared nearest-neighbour distance under $d_k$. Only $\alpha$ and $\beta$ are tuned. All tables cost one pass over a 4096-entry codebook per scale, which is trivial next to the tokenizer clustering cost that the DCPE paper worked to reduce.
Research Program
The work has three parts.
The first is descriptive. We compute $d_k$ for every scale of the public VAR tokenizer and ask how differently the scales rank the nearest neighbours of each code. The overlap of top-10 neighbour sets between the coarsest and finest scales is the key number. If rankings barely change across scales, the scale-aware idea reduces to a single better static metric, and we will report that and simplify the method.
The second is the scaling study. We train VAR at the depths supported by the ControlVAR codebase (12, 16, 20, 24, 30) under the plain objective and each alternative objective, using an identical data order and schedule. For each objective we fit
$$E(C) = a\,C^{-b} + E_\infty ,$$
where $E$ is an evaluation error, $C$ is training compute, $b$ is the scaling exponent, and $E_\infty$ is the floor. The power-law-plus-constant form follows the scaling fits VAR itself reports. The question is whether the distance-aware objective changes $b$, changes only $a$ (a constant compute multiplier), or changes only $E_\infty$.
One trap needs care. Training loss is not comparable across objectives, since the distance-aware objective is minimized against a different target. We therefore fit scaling curves on quantities every objective is judged on equally. The first is the expected decoded-feature error of the predicted token distribution under teacher forcing, which is $\mathbb{E}_{j\sim p_\theta}\,d_k(i,j)$ summed over scales. The second is plain top-1 token error. The third is FID, IS, and the full decomposition of cross-entropy.
The third part is diagnostic. What Scales in Cross-Entropy Scaling Law? splits cross-entropy into three terms using the rank of the correct token among the model's predictions: Error-Entropy, which scales as a power law, and Self-Alignment and Confidence, which it found do not scale and which come to dominate large language models. That analysis was done only on text. We apply the same rank-based decomposition to VAR scale by scale. This gives the first test of the decomposition on 2D multi-scale visual tokens, and it gives a mechanism test. If partial credit mainly reduces the non-scaling Confidence term at coarse scales, it is fixing a floor. If it speeds the decline of Error-Entropy, it is changing the slope.
Two lines of prior work supply the method; neither motivates it. Gaussian-smoothed targets over metric neighbours have worked across many settings: spatial label smoothing in TrajTok, Haversine smoothing over geographic cells in PIGEON, Gaussian-kernel coordinate cross-entropy in DocFusion, and position smoothing in DropPos. In each of these the output space comes with a natural metric: kilometres, pixels, or grid cells. Our contribution is to derive the metric from the generator's own decoding path. We also study how its benefit scales, which none of these papers examine.
Baselines
The do-nothing reference is VAR trained with plain cross-entropy at each depth, reproduced in our pipeline with our schedule. The strongest number in the file for this family is the reported ImageNet 256×256 FID of 1.73 from VAR at its largest configuration. The file gives no reported per-depth FIDs for depths 12 through 30, so we compare against our own reproduced runs and check our largest run against the 1.73 anchor. Beating this reference means a lower fitted $a$ or $E_\infty$ at matched compute, or a larger fitted $b$, with bootstrap confidence intervals over seeds and evaluation samples that do not overlap. It also means an FID gain at the largest depth larger than the seed-to-seed spread.
The strongest existing distance-aware competitor is DIST²Loss from Teaching Metric Distance to Discrete Autoregressive Language Models. It adds a discretized Gaussian target over codebook indices, weighted by cosine or Euclidean codebook distance, and reported FID 4.24 to 4.18 on LlamaGen at ImageNet 256×256. We port it to VAR unchanged: one static codebook distance at every scale, the same additive form, and the same $\alpha$ search as ours. This baseline doubles as the key ablation. The two methods differ only in whether distance is taken in raw codebook space or in the scale-specific decoded space, so any gap isolates the proposal's central ingredient.
The strongest non-smoothing competitor is DCPE from Exploiting Discriminative Codebook Prior for Autoregressive Image Generation. It groups codes with density-aware agglomerative clustering and injects the cluster structure as a prior, and it reported LlamaGen-B FID 5.29 to 4.83 and reached baseline FID 42% faster. We run DCPE's clustering on the VAR codebook and add the cluster-level supervision to VAR. We also run a hybrid in which clusters are built from $d_k$ instead of codebook distance, to separate "better metric" from "hard clusters versus soft targets".
Uniform label smoothing is the naive regularization baseline. For every method, $\alpha$, $\beta$, or $\epsilon$ is tuned by the same small grid at depth 16 only and then frozen for all depths. Tokenizer, data order, schedule, and compute are identical throughout. Beating a baseline means improving on it at the largest depth and having a fitted scaling curve that does not converge to it.
Why not just…?
Why not just use uniform label smoothing? It spreads target mass evenly over all wrong codes, so it softens overconfidence without saying which mistakes are near misses. DIST²Loss and DocFusion both report that spatially weighted targets beat undirected smoothing when outputs have geometry. Still, in images uniform smoothing could capture most of a Confidence-term benefit, because a large share of VAR's cross-entropy may be overconfidence rather than geometry. The rank-based decomposition settles this directly. If uniform smoothing lowers Confidence as much as our loss does and matches it on FID across depths, geometry is not doing the work, and we will say so.
Why not just apply DIST²Loss to VAR as published? This is the most serious threat, because it is our method with one choice changed. Its image result was small (4.18 versus 4.24). DCPE's analysis of codebook geometry gives a concrete reason to expect static codebook distances to be poor, but that is a hypothesis, not evidence. The head-to-head sweep and the cross-scale neighbour-overlap measurement decide it. If overlap is high and the two losses tie, the scale-aware part adds nothing, and the proposal reduces to a scaling study of DIST²Loss. That study would still be new, but it is smaller.
Why not just train a bigger model? If the gain from partial credit shrinks with depth, that is exactly the right advice. Our fits test it directly by reporting the compute multiplier at each depth. The cross-entropy decomposition paper argues that non-scaling loss terms eventually dominate, which suggests objective changes may matter more at scale, not less. That result comes from text, though, and we treat it as a hypothesis to test in images, not as support.
Why not just drop discrete tokens and use continuous autoregression? MAR reports FID 1.55 with a per-token diffusion head. EAR reports 1.97 with an energy-score head, though it needs a reduced learning rate on its generator head to stay stable. Both beat what we expect to reach. We do not claim to close that gap. Our goal is a cheaper improvement for discrete stacks, which keep the plain token interface that unified text-and-image models such as Liquid rely on. We report where our models sit relative to these continuous reference numbers without claiming parity.
Proposed Experiments
All experiments use class-conditional ImageNet-1K at 256×256 with the public VAR multi-scale tokenizer. The ImageFolder tokenizer suite serves as a second tokenizer for a robustness check at two depths. We evaluate FID and IS with 50k samples at fixed classifier-free guidance, which is also swept for the largest depth. We also report top-1 token error and expected decoded error per scale under teacher forcing, and the three-term cross-entropy decomposition per scale. Every configuration is trained with three seeds at depths 12–20 and one seed at depths 24 and 30.
The first decisive check is extrapolation. We fit scaling curves on depths 12 through 24 for each objective, predict the depth-30 decoded error and FID, and test the prediction against the held-out run. An objective whose advantage is real should keep its fitted ordering at depth 30. The second is the compute multiplier: the ratio of baseline compute to distance-loss compute needed to reach the same decoded error, reported at each depth. A flat or rising ratio supports the claim, and a ratio decaying toward one refutes it. The third is attribution by scale. Applying the distance loss only at coarse scales, and separately only at fine scales, tells us where the benefit comes from. It also tests the prediction that coarse scales, where one token covers a large area, gain the most. The fourth is inference-time error accumulation. We perturb a single coarse-scale token toward its nearest neighbour under $d_k$ and under codebook distance, and measure how much the finer scales recover in the final image. The distance matters only if the model's near-misses under $d_k$ really are recoverable.
References
- Teaching Metric Distance to Discrete Autoregressive Language Models
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
- Exploiting Discriminative Codebook Prior for Autoregressive Image Generation
- What Scales in Cross-Entropy Scaling Law?
- TrajTok: What makes for a good trajectory tokenizer in behavior generation?
- PIGEON: Predicting Image Geolocations
- DocFusion: A Unified Framework for Document Parsing Tasks
- DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions
- Autoregressive Image Generation without Vector Quantization
- Continuous Visual Autoregressive Generation via Score Maximization
- Liquid: Language Models are Scalable and Unified Multi-Modal Generators