Injecting Visual Tokens Where the Language Model's Geometry Opens Up

Vision-language models that pair a vision encoder with a large language model (LLM) usually feed the image in at the first layer, after passing it through a small projector, a linear layer or two-layer network that maps image features into the LLM's input space. Recent work suggests this is the wrong place. The LLM appears to spend its early layers doing nothing useful with the image. Visual tokens entering layer 1 already resemble the LLM's own mid-layer text representations. One system that skips the first eight layers entirely loses almost nothing. That system, however, chose its entry layer by grid search on a single backbone, and no one can say where the entry point should be for a different model.

We propose that the right entry depth can be read off the language model's own geometry before any multimodal training happens. Intrinsic dimension (ID) is the number of degrees of freedom a set of hidden vectors actually uses, as opposed to the thousands of coordinates they nominally occupy. In language transformers it rises from the input to a mid-depth peak, where surface form is discarded and semantic content is richest, and then falls as the model commits to an output. Our claim is that the best layer at which to insert visual tokens is the onset of this rise: the first layer where text representations begin expanding toward the peak. We further claim that a projector trained to match the neighborhood structure of text at that layer does better than one trained only through the usual next-token loss. Both claims can be tested causally, and the test also answers an open question about whether the ID peak does work or only accompanies it.

Core Claim

Three findings motivate the proposal. HiDrop finds that in LLaVA-1.5-7B, layers 0–8 are "passive propagators" whose text states ignore the image. Injecting visual tokens at layer 9 and removing them after layer 25 keeps 98.3% of baseline performance while pruning about 90% of visual tokens. LatentLens shows that visual tokens at layer 1 match contextualized text tokens from layers 8–16 and barely change with depth. Deep Pre-Alignment for VLMs calls the early-layer translation effort "wasted depth" and fixes it by placing a small language model in front of the large one.

A separate line of work supplies the proposed predictor. Emergence of a High-Dimensional Abstraction Phase in Language Transformers finds the ID peak across five 7–8B families, and finds that peak representations are highly mutually predictive across architectures. Geometry of Decision Making in Language Models shows that the peak moves earlier in larger models and precedes a "point of no return" in prediction. None of these papers connects the two observations.

The hypothesis is specific. Let $\ell^*$ be the injection layer that maximizes downstream multimodal accuracy, and let $\ell_{\text{on}}$ be the ID-onset layer. We define $\ell_{\text{on}}$ as the first layer whose text-only ID, estimated with the multiscale nearest-neighbor ratio estimator GRIDE, exceeds its early-layer baseline by half the rise to the peak. We predict that $|\ell^* - \ell_{\text{on}}|$ stays within one or two layers across backbones of different depth and scale. We also predict that a fixed-fraction-of-depth rule, or a rule copied from HiDrop, drifts as the peak moves.

The file contains one apparent contradiction. Exploring Intrinsic Dimension for Vision-Language Model Pruning reports flat, low ID (5–30) in the language layers of CLIP and BLIP, with no clear hunchback. Those are small text towers, not autoregressive decoders, so we do not expect the result to carry over. If a decoder backbone we study also shows a flat profile, the predictor is undefined for it, and we will report that as a limit of the method rather than hide it.

Research Program

The first stage is purely descriptive and cheap. For each frozen backbone we compute GRIDE and maximum-likelihood ID profiles on coherent text. We also compute profiles on shuffled text as a control, because the peak should vanish there, as the abstraction-phase paper found. The candidate backbones are Vicuna-7B (the LLaVA-1.5 base), Llama 3-8B, Mistral-7B, and Qwen3 at 0.6B, 1.7B, 4B and 8B. The Qwen3 series gives us depth and scale variation within one family. From each profile we read off $\ell_{\text{on}}$ and the peak layer.

The second stage makes the causal intervention the literature has not made. For each backbone we train LLaVA-1.5-style models under an identical data and compute budget, with the only difference being the layer at which projected visual tokens enter. We sweep every second layer from 0 to the peak. Skipped layers process text only, as in HiDrop's late injection. We do not prune tokens and do not use early exit, so that injection depth is the only variable.

This sweep gives the empirical $\ell^*$ per backbone. It also gives a direct test of whether putting information in before, at, or after the expansion phase changes what the model can use. That is the ablation-style evidence the abstraction-phase paper names as missing.

The third stage adds a geometric objective to the projector's alignment stage. The standard alignment stage trains only the projector, with the vision encoder and LLM frozen. We add a differentiable surrogate of information imbalance, the asymmetric nearest-neighbor rank statistic used in the abstraction-phase paper. The added term asks that neighborhoods among projected visual tokens predict neighborhoods among the paired caption's text states at the injection layer $\ell$:

$$\mathcal{L} = \mathcal{L}_{\text{NTP}} + \beta\,\Delta\big(P(v) \to h_\ell(t)\big),$$

Here $\mathcal{L}_{\text{NTP}}$ is the usual next-token loss. $P(v)$ is the projector output for image $v$. $h_\ell(t)$ is the layer-$\ell$ text representation of its caption. $\Delta$ is the imbalance, which is near 0 when the first space's neighbors predict the second's. $\beta$ is a weight tuned on held-out data.

This term is where the proposal differs from Deep Pre-Alignment. DPA buys alignment with an extra 0.6B-parameter perceiver. We ask whether a shallow projector, aimed at the right layer with a geometric target, reaches similar alignment at no inference cost. We also track, as a diagnostic, whether the ID of visual tokens at the injection layer lands near the text ID there.

Baselines

The naive reference point is standard LLaVA-1.5 injection at layer 0 with a two-layer MLP projector, as in MiniGPT-v2 and LLaVA. It is trained with exactly the data, steps and learning-rate budget of every other arm. The only number the file gives for it is that LLaVA-1.5 is "struggling to surpass 35%" on MMMU, per the MMMU paper. We will rerun it ourselves rather than quote it.

The strongest competitor on the depth question is HiDrop's late injection at a fixed layer 9. It is the only method that already moves the entry point, and it was tuned on exactly our Vicuna backbone. HiDrop's headline figure is 98.3% of baseline performance under about 90% token pruning, but that figure mixes pruning with injection. We therefore reimplement injection-only HiDrop on each backbone at layer 9. We also run a depth-proportional variant, layer 9 of 32 scaled to each model's depth. Beating HiDrop means two things. On Vicuna, the ID-predicted layer should match layer 9 within noise. On the other backbones it should beat both transplanted rules on the average of GQA, TextVQA, POPE, MMBench and MMMU, by more than the seed-to-seed standard deviation, measured over three seeds.

The strongest alignment competitor is Deep Pre-Alignment. It reports an average 1.9-point gain over eight multimodal benchmarks at Qwen3-4B. It also reports MATH-500 retention of 54.2, against 36.4 for the standard baseline and 84.8 for the base LLM. We compare on Qwen3-4B using DPA's eight-benchmark suite and its text-retention suite. The comparison counts as a win if ID-matched injection plus the imbalance loss reaches the same multimodal average at lower inference cost than DPA. DPA pays 2% of throughput at 32B, while our layer skipping should gain throughput. Separately, we check whether late injection reduces forgetting on MATH-500, since the early layers are never trained on visual input. No reported result was found in the file for DPA on MMLU-Redux or GPQA-diamond.

Two ablations isolate the key ingredients. Injecting at the ID-predicted layer without the imbalance term separates the choice of layer from the alignment loss. Injecting at the ID peak rather than its onset tests whether it is the onset specifically that matters.

Why not just…?

Why not just grid-search the injection layer, as HiDrop did? On a single backbone, grid search is the ground truth, and our sweep performs exactly that search. The problem is cost. Each point in the sweep is a full multimodal training run, while the ID profile is one forward pass over a few thousand text samples. The proposal is worth pursuing only if the cheap prediction matches the expensive search across backbones. The sweep decides this directly: if the predicted layer lands within the optimum's noise band on most backbones, the predictor has earned its place; if not, it has failed.

Why not just use a fixed fraction of depth? This is the most serious threat. If every model's useful window sits at the same relative depth, ID adds nothing. Geometry of Decision Making reports that larger models peak earlier in their stack, which predicts that a fixed fraction will drift across the Qwen3 size series. The drift may still be too small to matter, and the Qwen3 sweep is designed to tell us. If the fixed fraction ties with the ID rule, we will report that the geometry explains HiDrop's layer choice but is not needed to predict it.

Why not just pick the layer whose text tokens visual tokens already resemble, using LatentLens-style nearest-neighbor matching? LatentLens finds a broad band of matching layers (8–16) rather than a single one. The match also depends on a projector that has already been trained, so the rule is circular before training. We include it as a third predictor measured after the alignment stage and compare its error against the ID rule's.

Why not just add a perceiver in front, as DPA does? It works, and it is the benchmark to beat on accuracy. It does not, however, tell us where in the target LLM alignment should aim, and it adds parameters and latency. We test whether the geometric target recovers most of DPA's gain without those costs.

Proposed Experiments and Decisive Checks

The main result is a per-backbone plot of accuracy against injection layer, with $\ell_{\text{on}}$, the ID peak, HiDrop's layer 9 and the fixed-fraction rule marked on it. We summarize it as the mean absolute layer error of each predictor across the seven backbones.

Three further checks guard the interpretation. The first addresses a possible conflict with Understanding Information Storage and Transfer in Multi-Modal LLMs, which places multimodal factual recall in layers 1–4, so late injection may cut the image off from where facts are retrieved. We evaluate on VQA-Constraints and Encyclopedic VQA and report whether knowledge questions prefer earlier injection than perception questions. The second is the shuffled-text control. If shuffled-text profiles, which lack a peak, give predictions that are just as good, the effect is not about abstraction. The third addresses the unexplained second peak in OLMo-7B reported by the abstraction-phase paper. For any backbone whose profile has two peaks, we test injection at both onsets.

Success would give a training-free rule for designing vision-language models, and the first interventional evidence that the expansion phase is where a transformer becomes able to take in new abstract content.