Factorized Visual Codes and the Compositional Failure of Discrete Generators

Discrete image models first turn a picture into a short sequence of integer "visual words" using a learned dictionary, a step called vector quantization. They then treat image generation like language modeling: predict the next word. This makes it easy to put images and text in one model. A careful study has shown a real cost, though. When asked to produce an unseen combination of familiar attributes, such as a shape they have seen in a color they have also seen but never together, these discrete models fail far more often than continuous ones (What Drives Compositional Generalization in Visual Generative Models?). Other work points the other way. Discrete codes split into separate attribute slots support near-perfect out-of-distribution reasoning on synthetic scenes (Object-Centric Semantic Vector Quantization), and quantizing word embeddings into structural classes makes Transformers systematic on SCAN (Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings).

We propose that both findings follow from one property: whether a novel combination looks like a new symbol or like a new arrangement of old symbols. Suppose each image patch gets one index from a single joint dictionary. Then "red sphere" and "blue cube" are separate classes, and "blue sphere" is a class the model has never been trained to output. Now suppose each patch gets a tuple of indices from small dictionaries, one per factor. Then "blue sphere" is a tuple whose parts were all seen in training. We will test this directly. Our approach is to make the tokenizer's factors line up with the words that describe them, so that the text prompt and the image tokens share the same compositional units. That is where the cross-modal gap enters.

Core Claim

The compositional failure of discrete visual generators comes from joint codebooks, not from discreteness. Hold the bits per patch, the generator, and the training budget fixed. A generator over factorized codes will close most of the gap to a continuous model on held-out attribute combinations. A generator over joint codes will not. The size of the failure on a given test image should be predicted by a simple count: the fraction of its tokens that are unseen symbols, as opposed to unseen co-occurrences of seen symbols.

This count is a visual version of the unobserved-local-structure analysis in Unobserved Local Structures Make Compositional Generalization Hard. That paper predicted failures on semantic-parsing examples from their unseen small subgraphs, with AUC 78.4 to 93.3.

The closest prior work does not settle the question. Farid et al. interpolate between a masked discrete generator and a continuous diffusion transformer. They blame the categorical objective and fix it with an auxiliary continuous prediction loss, but they never vary how the codebook is factored. FQGAN (Factorized Visual Tokenization and Generation) and LooC (LooC: Effective Low-Dimensional Codebook for Compositional Vector Quantization) factorize codebooks to improve reconstruction and codebook usage, but they never test recombination. SVQ factorizes object slots by attribute, but it has no text and no conditional generation. UniCode (UniCode: Learning a Unified Codebook for Multimodal Large Language Models) shares one vocabulary across image and text, but it does not factor it and was never evaluated on compositional splits.

Research Program

We will build one text-conditioned masked generator in the MaskGIT style. It predicts masked image tokens from the caption and the visible tokens. We will pair it with four tokenizers that share an encoder and decoder architecture and use the same number of bits per patch:

  • a joint codebook, for example 4,096 entries (12 bits);
  • a product code with arbitrary channel splits, in the LooC style, for example three sub-codebooks of 16 entries;
  • a factorized code whose sub-codebooks are pushed toward independence by the orthogonality regularizer of FQGAN;
  • a caption-anchored factorized code.

In the caption-anchored code, each sub-codebook gets a small contrastive loss tying its index distribution over an image to one slot of caption words: color words, shape words, material or texture words. The model learns which factor a caption word belongs to from co-occurrence alone. It never sees attribute labels. This is our cross-modal ingredient. A prompt word and the image sub-token it controls become nearly one-to-one, so recombining words in the prompt maps directly onto recombining sub-tokens.

The analysis tool is the unseen-symbol count. For every test image we encode it and record two fractions. The first is the share of patches whose token, or token tuple, never appeared in the training set. The second is the share whose individual sub-tokens are all familiar but appear together for the first time. We then measure how well each fraction predicts generation errors, as AUC over test images, following Bogin et al. Our account predicts that joint codes push held-out combinations into the first category and factorized codes push them into the second, and that the first category drives failure.

Baselines

The naive reference point is the joint-codebook masked generator with no extra loss. This is the setting Farid et al. found to fail on held-out attribute pairs ("Level-2" compositions in their terminology).

The strongest competitor on the scientific question is the continuous diffusion transformer used in that same paper, trained on the same continuous latents before quantization. The file does not report its Shapes2D or Shapes3D numbers. On CoVLA compositional retrieval, Farid et al. report that continuous models beat categorical ones by +161% to +207%. We will rerun this baseline ourselves rather than borrow a figure.

The strongest fix inside the discrete family is the joint codebook plus Farid et al.'s auxiliary continuous prediction loss (a JEPA-style loss, predicting continuous features of masked patches). For SugarCrepe-style evaluation, the strongest reported reference is GPT-4V at 90.95% overall and 83.13% on SWAP (SugarCrepe). We cite it only to set the ceiling. It is not a matched competitor.

All runs will use the same data and generator size. Each method gets the same optimizer steps and the same hyperparameter sweep budget, for example eight configurations chosen on in-distribution validation loss. No configuration will be selected using held-out combinations. We will use five seeds and report confidence intervals.

Our proposal beats the field in two cases. In the first, caption-anchored or orthogonal factorized codes come within the seed noise of the continuous model on held-out-combination accuracy while the joint code does not. In the second, factorized codes without the continuous auxiliary loss match the joint code with it. The arbitrary-split product code is the ablation that isolates our key ingredient. If it does as well as the aligned factors, then "any product code" is the answer and semantic alignment is unnecessary.

Why not just…?

Why not just add the continuous auxiliary loss, which already works? This is the main threat. Farid et al. show it helps discrete generators, and if it closes the whole gap, factorization adds little for generation. Our claim differs on mechanism: the auxiliary loss repairs a symptom that factorization removes at the source. The direct test crosses the two treatments: joint versus factorized codes, each with and without the auxiliary loss. If factorized codes gain nothing from the auxiliary loss while joint codes gain a lot, the mechanism is the code structure. If both gain equally, we will report that factorization is not sufficient.

Why not just use continuous tokens and skip the codebook? Continuous models do generalize better in Farid et al.'s comparison, and we include them as the target. The question is not whether continuous models work. It is whether unified models that speak in discrete tokens, as UniCode and BEiT-3 do (Image as a Foreign Language), must pay a compositional tax. Continuous high-capacity interfaces are also not automatically systematic. In CLOSURE, module networks passing spatial tensors scored 25 to 64%, while compressed vector interfaces scored 86 to 98%.

Why not just use an existing factorized tokenizer like FQGAN or LooC unchanged? This is plausible, so both are run as conditions. FQGAN's factors separate semantic level from texture, not attribute from attribute. LooC's splits are arbitrary channel groups, so neither guarantees that a novel attribute pair becomes a recombination of seen sub-tokens. The unseen-symbol count will show which encodings actually make held-out combinations look familiar. If LooC-style splits already do, we will drop the caption anchoring and say so.

Why not just scale or fine-tune with hard negatives? CREPE reports that larger pretraining corpora still leave up to a 12% relative Recall@1 drop on unseen compounds (CREPE). SugarCrepe shows that hard-negative fine-tuning (NegCLIP) mostly exploited artifacts in earlier benchmarks. Neither approach changes what a held-out combination looks like to the model.

Proposed Experiments

The decisive experiment is controlled and synthetic. On Shapes3D and on CLEVR-Tex we will hold out a set of attribute pairs, each of whose single attributes appears in training, and generate from captions that name the held-out pairs. Correctness will be scored by attribute classifiers trained on held-out ground-truth renders. We will also report rFID, so that no factorized code wins simply by reconstructing worse. For context on CLEVR-Tex, SVQ reports a generation FID of 84.52 against 178.59 for VQ-VAE.

Our account makes two predictions. First, joint codes put a large share of held-out-pair patches into the unseen-symbol category while factorized codes put almost none there. Second, unseen-symbol share predicts per-image failure better than unseen-pair share. A failure of either prediction falsifies the account even if factorized codes happen to score higher.

To check that the effect survives real images, we will train the same tokenizer variants on MSCOCO captions and evaluate in two ways. The first is the unseen-pairs protocol, where COMiT-L reports 45.31% top-5 (Communication-Inspired Tokenization for Structured Image Representations). The second is SugarCrepe, run as caption scoring by the generator's conditional likelihood of the image tokens. Here caption anchoring should matter most on SWAP and attribute-replacement items. SugarCrepe's adversarial filtering means a text-only scorer sits at chance, so any gain must come from the image tokens.

The cross-modal test is whether anchored factors raise SWAP accuracy relative to orthogonal-only factors at matched reconstruction quality. If they do not, the cross-modal component is decoration. In that case the result reduces to a cleaner and still useful claim: how a codebook is factored, rather than whether it is discrete, determines compositional generalization.