Drowned, Not Erased: Spectral Conditioning of the Vision-to-Language Projector

Vision-language models attach a pretrained image encoder to a large language model (LLM) through a small "projector," usually a two-layer neural network that turns each image patch feature into a vector the LLM reads as if it were a word. The projector is trained only to help the LLM predict captions and answers. Domain adaptation research has studied a closely related pressure for years. When a network is pushed to make two distributions look alike, it tends to keep a few dominant feature directions and suppress the weaker ones, and those weaker directions are often the ones that separate fine-grained classes. We propose that the projector faces the same trade-off between matching the language side and keeping visual detail. We also propose that the typical failure is drowning rather than erasure. The visual detail is still in the projector's output and can still be recovered, but it sits in low-variance directions that the LLM effectively ignores, while a few high-variance directions near word embeddings dominate.

This distinction decides which fix is appropriate. If detail is erased, the remedy is to add information back, for example by reconstructing the image. If detail is drowned, the remedy is cheaper: reshape the spread of variance across directions at the projector's output. We will first measure which failure occurs. If drowning dominates, we will test a regularizer that matches the shape of the projector's output spectrum to the spectrum of the LLM's own word embeddings. This regularizer is designed to leave the mean offset between modalities untouched.

Core Claim

During standard projector training, the singular value spectrum of projected visual tokens becomes concentrated. We predict that this concentration, not information loss, explains much of the projector's contribution to object hallucination and to fine-grained or cross-domain perception errors. Concretely, a linear probe on projector outputs should recover nearly everything a probe on raw encoder features recovers. The LLM's internal representations of those same tokens should not, and the gap should track how concentrated the projector spectrum is. Correcting the spectrum's shape, without forcing visual tokens onto text tokens, should reduce hallucination at negligible cost.

Grounding in Prior Work

The mechanism comes from domain adaptation. Batch Spectral Penalization (Chen et al.) found that adversarial alignment lets a few top singular directions dominate the feature matrix, and that penalizing them restores class-discriminative secondary directions. This motivates our hypothesis, and its top-$k$ penalty is our main spectral baseline. Its known weakness is that $k$ must be tuned by hand, which our target-spectrum formulation avoids. Several other papers establish that full invariance destroys task-relevant structure: Lu et al., Shi et al., Wang on fine-grained "information collapse," and Zhang et al.. All of them study classifiers aligned across image domains, not projectors aligned across modalities. Disentangling approaches such as Kong et al. and DA-Ada's dual adapters (Li et al.) handle the trade-off with extra branches. We deliberately avoid extra branches and act on the single projection interface.

Two papers are closest in spirit. Chaudhry shows that self-supervised projection heads deliberately crush nuisance directions, compressing rotation orbits 21.85x, to absorb invariance pressure away from the backbone. That analysis covers single-modality contrastive learning. In a vision-language model, by contrast, the projector is the last stop before the consumer, so any crushing it does is passed straight to the LLM. Li et al. show "semantic entanglement" in LLaVA-1.6's projector: logo images map onto brand-name token directions. Zeroing the top 32 predictive dimensions cut logo hallucination by 29.6% at a 3.2% OCR cost. That is a post-hoc ablation found with a probe. We ask whether a training-time spectral constraint prevents the entanglement from forming.

A third paper constrains the design. Liang et al. show that the gap between image and text embedding regions in contrastive models is functional: closing it fully hurt zero-shot accuracy, while tuning it raised CIFAR-10 top-1 from 0.9013 to 0.9081. Our regularizer therefore matches spectral shape and explicitly leaves the mean offset alone.

Two further papers supply competing remedies. ROSS (Reconstructive Visual Instruction Tuning) adds a denoising head that reconstructs frozen-VAE latents from LLM outputs. This assumes erasure and adds a teacher and a diffusion head. Deep Pre-Alignment (Yu et al.) routes visual features through a 0.6B-parameter language model before the target LLM. REPINA (Razdaibiedina et al.) contributes a recoverability penalty: pretrained features must remain recoverable through a learned map. We use it as the "anti-erasure" control.

Research Program

Diagnosis. We train the LLaVA-1.5 recipe (Liu et al.), consisting of a CLIP-ViT encoder, a two-layer MLP projector, and Vicuna-7B. We save checkpoints throughout projector pretraining on LLaVA-558K and through instruction tuning. At each checkpoint we measure three quantities on held-out images:

  • Spectral concentration. The effective rank and normalized eigenvalue profile of projected tokens.
  • Recoverability. Linear-probe accuracy for fine-grained and domain-shifted labels, using Office-Home and VisDA-2017 classes and LogoDet-3K text-versus-symbol labels. Probes are fit on raw encoder features, on projector outputs, and on LLM hidden states at several depths.
  • Vocabulary snapping. The mean cosine similarity of projected tokens to their nearest word embedding, which follows Li et al.'s entanglement finding.

Interpretation follows directly from where accuracy falls. "Drowned" means probe accuracy is preserved from encoder to projector but falls inside the LLM, and the fall correlates with concentration. "Erased" means probe accuracy already falls at the projector. These three measurements are cheap and are a deliverable in their own right.

Intervention. Let $Z$ be the centered matrix of projected visual tokens in a batch. Let $p_i = \sigma_i^2(Z) / \sum_j \sigma_j^2(Z)$ be its normalized eigenvalue profile. Let $q_i$ be the same profile computed from the LLM's input embeddings of text tokens, either from the batch's own answer tokens or from a fixed vocabulary sample. We add

$$ \mathcal{L} = \mathcal{L}_{\text{text}} + \lambda \, \mathrm{KL}(q \,\|\, p), $$

where $\mathcal{L}_{\text{text}}$ is the usual next-token loss and $\lambda$ is a scalar weight. Centering removes the mean, so the gap between modalities is untouched; the penalty only asks visual tokens to spread their variance across directions the way words do. It has no $k$ to choose, and it costs one SVD of a batch-by-dimension matrix per step. Because the text spectrum is itself somewhat anisotropic, matching it is a different hypothesis from plain whitening, and we test that difference directly.

Baselines

The naive reference point is the unmodified LLaVA-1.5-7B recipe with its MLP projector. On it, HalluTrace reports a POPE-adversarial F1 of 85.9% and a COCO CHAIR₁ of 14.2. We retrain this baseline ourselves on identical data, with identical seeds and compute, so every comparison is internal.

The strongest training-time competitor for detail retention is ROSS. It reports 57.1 on HallusionBench and 56.7 on MMVP, but under a larger instruction mix (Cambrian-737K), so those numbers are not directly comparable. We will rerun ROSS on our data mix with its published head sizes and compare at matched wall-clock. Our method beats it if it matches ROSS on POPE-adversarial, MMVP and HallusionBench within seed variance at a fraction of the overhead, or exceeds it on fine-grained and logo evaluations.

The direct domain-adaptation analogue is BSP applied to projector outputs. For BSP we sweep $k \in \{1, 2, 4, 8\}$ and give it the same $\lambda$ search budget we give our method, five values. Beating it means matching its best-$k$ result without searching over $k$, or exceeding it, which would show that targeting the text spectrum adds something beyond suppressing top directions.

The anti-erasure control is a REPINA-style term in which a jointly trained linear map must reconstruct frozen encoder features from projector outputs. If the diagnosis says "drowned," this term should help little, because the information is already recoverable. If it helps as much as spectral matching, the drowning story is wrong.

Two ablations isolate the key ingredient. In the first, the target $q$ is replaced by a uniform profile, which amounts to plain whitening; this asks whether the text spectrum specifically matters. In the second, an added term also pulls the visual mean onto the text mean, following Liang et al.; we predict this hurts. Hallucination-Aware Decoding from HalluTrace (POPE-adversarial F1 89.6%, CHAIR₁ 8.1) is reported as an orthogonal inference-time method, and we also report its combination with our projector.

Why not just…?

Why not just raise image resolution? Higher resolution adds pixels, but it does not change how the projector distributes variance. Li et al. find that high-resolution frontier models still label only 45–47% of pure-symbol logos correctly as containing no text. We will run our regularizer at 336px, as LLaVA-1.5 already does. The question is whether the gain survives at that resolution, not whether it replaces resolution.

Why not just use a bigger or deeper projector? The move from linear to MLP is already in our baseline. The spectral diagnosis will show whether added capacity reduces concentration or worsens it. Chaudhry's finding that nonlinear heads induce singularities predicts that it worsens. We include a three-layer projector arm to check.

Why not just add a LayerNorm or whitening step to the projector output? This is the most serious threat. If uniform whitening equals text-spectrum matching, our contribution reduces to normalization. The uniform-target ablation settles this. If it ties, we will say so and report the cheaper method as the finding.

Why not just use ROSS? If the diagnosis shows erasure at the projector, ROSS-style reconstruction is the right tool and our spectral term should lose to it. That outcome would be a clean negative result. Under drowning, ROSS spends a teacher model and a diffusion head to fix a conditioning problem that a single SVD-based penalty can address. The head-to-head comparison at matched compute decides between these readings.

Why not just fix it at decoding time? Decoding interventions such as HAD add multiple forward passes, are tied to fixed object vocabularies, and leave the projector unchanged. We report them as complementary, not as substitutes.

Proposed Experiments

All arms share LLaVA-558K pretraining and LLaVA-Instruct-158K tuning on a Vicuna-7B backbone, with three seeds each. Hallucination is measured with POPE (random, popular and adversarial splits) and CHAIR₁ on COCO. Perception is measured with MMVP, HallusionBench, MME and MMBench. Entanglement is measured on LogoDet-3K and Hard-60, using Li et al.'s 29.6% reduction from dimension ablation as a reference point. Fine-grained cross-domain perception is measured by asking the model to name Office-Home and VisDA-2017 categories across domains, alongside the probes described above. We also run a secondary arm on a Qwen3-4B LLM with MATH-500 to measure language forgetting. Yu et al. report MATH-500 falling from 84.8 to 36.4 under standard tuning and recovering to 54.2 with DPA, but on their own recipe, so we rerun the baseline rather than compare across setups.

The decisive checks are three:

  • The diagnosis must show encoder-to-projector probe accuracy roughly preserved while LLM-layer accuracy falls in step with spectral concentration. If instead accuracy drops at the projector, we pivot to the recoverability term and to ROSS-style remedies.
  • Spectral matching must improve POPE-adversarial F1 and logo hallucination over the retrained baseline beyond seed variance, without lowering MME or MMBench.
  • Text-spectrum matching must beat uniform whitening and best-$k$ BSP. Otherwise the claim narrows to "conditioning matters," with the cheapest effective conditioner as the recommendation.