Binding by Construction: Matrix Operators Inside the Visual Projector

Multimodal language models see an image through a small learned layer, the projector, that turns a vision encoder's patch features into "soft tokens" the language model reads as if they were words. These projectors are flat: each patch becomes a token, and nothing in the design says which colour belongs to which object. The result is a well-documented failure. These models often cannot tell "a red car and a blue truck" from "a blue car and a red truck." This proposal borrows an old idea from compositional semantics, that nouns are vectors and adjectives are matrices acting on them, and builds it into the projector. Each object in the scene becomes an argument vector. Each visual attribute becomes a low-rank matrix drawn from a bank of operators fitted on the language model's own text representations. The token the language model receives is the product of the two, so the visual "red" is literally the same linear map as the textual "red," and it is applied only to the object it describes.

The claim is narrow and testable. If attributes enter as shared operators rather than as features mixed into a patch, the model should generalize to attribute–object pairings it never saw during training, and it should resist swap-style probes that fluent-caption benchmarks have shown loss tricks cannot fix. It should do this without the language forgetting that flat projectors cause. The main competitor is not a flat projector. It is a projector with the same object-level supervision but no factorization, and the experiments are built around beating it on held-out combinations.

Where the two directions meet, and what is already done

The matrix-vector tradition began with Nouns are Vectors, Adjectives are Matrices. That work estimated one matrix per adjective by regression from noun vectors to observed phrase vectors and found it beat additive and multiplicative composition. An Empirical Study on Compositionality in Compound Nouns showed that symmetric arithmetic misses head–modifier asymmetry, and Studying the Recursive Behaviour of Adjectival Modification showed that stacked adjective matrices encode ordering constraints. The estimation recipe is the one implemented in DISSECT, which we reuse. This line supplies our method. It also supplies a constraint: one fixed matrix per word handles polysemy poorly. We address that by predicting each operator from visual evidence rather than looking it up.

On the projection side, Multimodal Few-Shot Learning with Frozen Language Models established the "visual prefix," in which projected image features are placed before the text prompt. MiniGPT-v2 and Video-LLaVA refined it with single MLP projectors. Deep Pre-Alignment for VLMs motivates the present work most directly. It showed that shallow projectors leave visual tokens far from text space, wasting early language-model layers and causing forgetting: MATH-500 falls from 84.8 to 36.4 for a 4B LLaVA-style model and recovers only to 54.2 with its 0.6B perceiver front-end. That perceiver does the alignment as a black box, and the paper never evaluates attribute binding. Vision Language Models Map Logos to Text via Semantic Entanglement in the Visual Projector places entanglement in the projector itself, which is where we intervene.

Three papers come closest to the intersection, and each leaves the specific gap open. DisCoCLIP builds the text encoder of a CLIP-style retrieval model as a grammar-driven tensor network and reports 93.68% on SVO-Swap. Its image, however, stays a single frozen vector, so all the structure lives on the language side. We put the operators on the vision side, inside an autoregressive model. ComAlign gets binding gains (+5.55% on VG-Attribution over CLIP) from detector boxes and a small transformer. It uses no algebraic operators, depends on YOLOv9 proposals at test time, and works only for retrieval. Doc-to-LoRA turns visual activations into a weight update for a text-only model. That update is a single unstructured low-rank delta rather than per-constituent operators. To our knowledge, no prior work makes visual tokens out of operator–argument products whose operators are shared with the language model's text representations.

Core Claim

Factoring visual object tokens as the product of an argument vector and an operator taken from a text-fitted operator bank gives three things that an equally supervised flat projector does not. It gives higher accuracy on attribute–object combinations absent from training. It gives better attribute binding on fluent, adversarially filtered swap probes. It gives operators that can be edited causally: replacing one object's "red" operator with "blue" changes only that object's description. Language forgetting should stay no worse than a standard projector.

Research Program

The model keeps a standard two-layer MLP path for patch tokens, so global and texture information is not lost. Beside it runs a compositional path. A small set of learned query slots cross-attends over the vision encoder's patches. Each slot $k$ produces an argument vector $v_k$ and mixing weights $\alpha_{kj}$ over a bank of $J$ attribute operators. The slot's token is

$$z_k = \Big(I + \sum_{j=1}^{J} \alpha_{kj}\, U_j V_j^{\top}\Big)\, v_k ,$$

where $U_j V_j^{\top}$ is a rank-$r$ matrix for attribute $j$ and $I$ is the identity, so an object with no salient attribute passes through unchanged. Stacked attributes are applied as an ordered product of such factors, which lets us test the ordering effects documented by Vecchi et al.

The operator bank is fitted before any visual training, on text alone. For each frequent Visual Genome attribute, we feed the frozen language model phrases such as "red car" and "car" across many nouns. We take its hidden state at the phrase's last token at a fixed early layer $\ell$. We then estimate a low-rank $U_j V_j^{\top}$ by regularized least squares from noun states to phrase states, as in the DISSECT recipe. Slot tokens $z_k$ are injected at that same layer $\ell$, which is a swept hyperparameter, so visual tokens enter a space the operators were fitted in. During projector training the bank is frozen, with an optional small learned residual per operator. This keeps "red" tied to its textual meaning while letting the mixture weights handle visual sense variation, which is the polysemy problem that fixed matrices cannot handle.

Supervision for the compositional path comes from Visual Genome region annotations. Slots are matched one-to-one to annotated regions by minimum-cost assignment. The matched slot is trained in two ways. Its $v_k$ is pushed toward the language model's layer-$\ell$ state for the bare noun. Its $z_k$ is pushed toward the state for the full attribute-plus-noun phrase. Boxes are used only to compute this training-time matching; at test time no detector runs, which avoids ComAlign's dependence on proposals. The overall recipe follows the DPA two-stage schedule (558K caption pairs for alignment, then 1M instruction samples) on Qwen3-4B, with the Visual Genome region objective added during stage one. Binary relations are a second phase. There, a relation operator is generated from the object slot and applied to the subject slot, so "horse eating grass" and "grass eating horse" produce different products by construction.

Baselines

The naive reference point is a LLaVA-style model: frozen vision encoder, two-layer MLP projector, Qwen3-4B, trained on exactly the same 558K + 1M data. It anchors every metric and the forgetting measurement. DPA reports 36.4 MATH-500 for this configuration against 84.8 for the base model. We have found no reported result on Lacuna for such a model on SugarCrepe or ARO, so we measure that ourselves.

The strongest existing method is Deep Pre-Alignment. We reimplement its 0.6B perceiver front-end on the same data and target model. DPA reports +1.9 points averaged over eight multimodal benchmarks at 4B and 54.2 on MATH-500 (Deep Pre-Alignment for VLMs). To make the comparison fair, DPA also receives the Visual Genome region captions as extra instruction data. Beating it means a clear gain on the SugarCrepe swap subsets and on the held-out-combination split, with general-suite and MATH-500 scores within noise of DPA's.

The ablation that isolates the key ingredient is a region-supervised flat projector. It has the same slots, the same Hungarian matching and the same phrase-state targets, but each slot emits $z_k$ through an MLP rather than an operator product. Parameter count and tuning budget are matched; every variant gets the same hyperparameter sweep over learning rate and the injection layer $\ell$. Two further ablations separate the parts of the idea. One is an additive version, $z_k = v_k + a_k$, which tests the matrix versus addition question within vision. The other is a free-operator version whose bank is learned from scratch rather than fitted on text, which tests whether sharing operators with language matters. The proposal succeeds only if the full model beats the flat region-supervised ablation on held-out combinations.

A hard-negative baseline trains the naive MLLM with swapped-caption negatives through a likelihood-ranking loss. This tests whether the loss trick that inflated NegCLIP's ARO numbers carries over to generative models.

Why not just…?

Why not just supervise a flat projector on region phrases? This is the real threat. The language model's phrase representations already encode much of compositional structure, so an MLP trained to hit them might generalize to unseen pairs without any factorization. Nothing in the file settles this. The region-supervised flat ablation is therefore the decisive comparison. If it matches the operator model on the held-out-combination split, we would conclude that the supervision, not the algebra, does the work, and we would report that.

Why not just train with hard negatives? When and Why Vision-Language Models Behave like Bags-Of-Words reported NegCLIP raising VG-Relation from 63% to 81% and COCO-Order from 46% to 86%. SugarCrepe then showed that blind text models beat vision models on 9 of 10 older compositional tasks, and that NegCLIP's gains vanish on fluent negatives. We include a hard-negative baseline and evaluate it only on SugarCrepe so that it cannot win through text artifacts.

Why not just use a bigger or deeper front-end like DPA? DPA reduces forgetting. However, SugarCrepe reports that GPT-4V, presumably far larger, still reaches only 83.13% on swap items against roughly 99% for humans, which suggests capacity alone does not fix binding. DPA is run as a matched baseline, so this question is settled directly rather than argued.

Why not just use detector-based alignment like ComAlign? ComAlign improves retrieval but needs YOLOv9 boxes at inference, leaves relation direction unresolved by its authors' own account, and does not feed an autoregressive model. Our slots learn from boxes only during training. We report SugarCrepe results separately for relations without a compact spatial footprint, where detector crops are weakest.

Proposed Experiments

The primary benchmark is SugarCrepe, whose adversarial filtering puts blind text models at 50%. We report the swap-attribute, swap-object, replace-attribute and replace-relation subsets. Because SugarCrepe is a retrieval benchmark, each candidate caption is scored by its length-normalized log-likelihood given the image, and as a check also by two-choice prompting. The ceiling references are GPT-4V's 90.95% overall and 83.13% on swaps. ARO's VG-Attribution and VG-Relation and the 95-pair SVO-Swap set (where DisCoCLIP reports 93.68%) serve as secondary probes. We discount them where SugarCrepe shows they are hackable.

The decisive new evaluation is a held-out-combination split. We withhold a few hundred attribute–noun pairs, such as "striped" with "car," from all Visual Genome training regions while keeping each word seen with other partners. We then build two test sets on these pairs: fluent swap probes in the SugarCrepe style, and an open-ended "describe the object at this location" generation task scored for attribute correctness. The prediction is that the operator model's advantage over the flat region-supervised ablation is larger on held-out pairs than on seen pairs. A gap that appears only on seen pairs would mean the factorization adds nothing.

Two mechanistic checks follow. In the operator-swap intervention, we replace one slot's operator mixture with that of another attribute at inference and measure how often the generated description changes the target object's attribute while leaving other objects untouched. The flat ablation gets the matched intervention of swapping MLP outputs between slots. For stacking order, we compare $M_{a} M_{b} v$ with $M_{b} M_{a} v$ on multi-attribute regions and test whether the preferred order matches the language model's own text-side preference.

Finally, we run the guardrails. These are the eight-benchmark DPA multimodal suite (MMMU, MathVista, MathVision, OCRBench, AI2D, SeedBench2Plus, MM-Vet, MMStar) and MATH-500 forgetting. The operator path must not cost general performance or increase forgetting relative to the naive baseline, and ideally should approach DPA's 54.2 at much lower added cost.