Does Context Extension Quietly Break Visual Composition?

Vision-language models are increasingly stretched to read hundreds of thousands of tokens of video, image sequences or long captions. The standard reassurance is that this costs little on ordinary short inputs: papers report near-lossless scores on familiar short benchmarks after extension. Those benchmarks share two weaknesses. They test coarse recognition rather than whether the model knows which object is red or which thing is on top. They are also built from web images that models have probably seen during training. We propose to test whether context extension degrades fine-grained compositional understanding ("the red cube left of the blue sphere" versus "the blue cube left of the red sphere") in a way that static, possibly memorized benchmarks hide.

The method pairs two kinds of measurement on the same models. The first is the usual static minimal-pair benchmarks. The second is a procedurally generated twin: freshly rendered scenes whose ground-truth layout is known, which cannot have been in any training set. Both are run on matched pairs of models, one before and one after context extension. The quantity of interest is not absolute accuracy, since synthetic scenes are harder for everyone. It is the difference between how much the extended model loses on fresh scenes and how much it loses on static ones. A large gap would mean that current "near-lossless" claims rest partly on memorized associations. A specific ablation then tests whether the damage comes from compressing the spatial axes of the position encoding.

Core Claim

The hypothesis is that position-encoding changes used for context extension selectively weaken the attention patterns that bind attributes and relations to image regions. The first such change is interpolation, which squeezes long position indices into the range seen in training. The second is training the model to tolerate perturbed positions. Memorized subject–relation–object associations from web data survive these changes and prop up static benchmark scores.

The claim makes three falsifiable predictions. First, the extension-induced drop is larger on procedurally generated pairs than on static pairs drawn from the same relation types. Second, the drop concentrates on spatial relations and on rare or inverted triplets, such as "grass eating a horse," rather than on frequent ones. Third, leaving the spatial position axes unscaled recovers most of the loss. If the static and procedural drops come out equal, the masking hypothesis is wrong, and existing static benchmarks are adequate for auditing extension. We would report that outcome as a result.

Prior Work and Where This Sits

Motivation from context extension. Three lines of extension work motivate the question and supply the treatments.

  • LongRoPE and LongRoPE2 rescale rotary position embeddings (RoPE). RoPE encodes position as rotations at many frequencies, and these papers rescale it to reach 128K to 2M tokens. They argue that stretching dilutes the high-frequency components used for local distinctions. They check short-context retention only on static text benchmarks.
  • GIRAFFE extends vision-language models to 128K tokens. It uses M-RoPE++, which interpolates along the image height and width axes and extrapolates along time. It also keeps 40% short multimodal data in the training mix. It checks retention on MME and MMBench but not on minimal pairs.
  • Long-CLIP stretches CLIP's text positions while freezing the first 20. It preserves ImageNet zero-shot accuracy, but it never isolates relational binding.

All three leave unexamined exactly the measurement we propose.

Mechanistic evidence that constrains the design.

  • Lin et al. show, on uncontaminated synthetic functions, that length-extrapolating position encodings disrupt induction heads. These are attention circuits that copy patterns seen earlier in the context. That work is one-dimensional and text-only.
  • Understanding Synthetic Context Extension via Retrieval Heads finds that extension recruits a narrow set of retrieval heads.
  • Shuffle the Context deliberately trains invariance to perturbed RoPE indices. This sits in direct tension with the "bag of words" diagnosis of compositional failure in ARO / NegCLIP and Winoground, where models are already too indifferent to order.
  • CompoDistill localizes relational failure to misaligned visual attention in the middle 30–70% of language-model layers. That tells us where to look, but CompoDistill never varies context length.

Methods we reuse from contamination work.

  • DyVal generates reasoning problems on the fly. It shows that models strong on static benchmarks (phi-1.5, WizardMath-13B) "scored nearly zero" on fresh ones.
  • A²RBench verifies generated tasks programmatically. It uses symbolic remapping to expose a 17.7% drop attributable to familiar token formats.

Both are text-only. We carry their two ideas, fresh generation and remapping controls, into rendered visual minimal pairs.

Metric choice from test-time matching. Test-Time Matching argues that the standard GroupScore understates competence. GroupScore counts an item correct only if both images and both captions match correctly. Test-Time Matching instead proposes GroupMatch, a global one-to-one assignment, plus test-time fine-tuning on pseudo-labels. We adopt GroupMatch as a second metric so that any degradation we find cannot be dismissed as an artifact of the scoring rule.

Research Program

The generator. The first artifact is a minimal-pair generator. Each item is a scene graph: objects with shape, color, size and material, plus pairwise spatial relations (left of, above, in front of, inside) and a small set of depicted actions. A programmatic renderer turns the graph into an image. The same graph is turned into a caption by a template, and a second, swapped graph yields the paired image and caption, following the two-images, two-captions structure of Winoground.

Because the scene graph is known, verification is exact. We re-derive relations from rendered object coordinates and discard any pair whose swap is visually ambiguous. This is the visual analogue of A²RBench's round-trip check.

Two knobs give the generator its power:

  • A frequency knob. Triplets are sampled either from frequent Visual Genome subject–relation–object combinations or from rare and inverted ones. This lets us read contamination-like reliance on priors directly from the accuracy gap.
  • A remapping knob. Familiar color and shape names are replaced with novel invented shapes and arbitrary named palettes, following A²RBench's symbolic remapping.

The length axis. The second component places each minimal-pair query after a controlled amount of preceding context: zero, 4K, 32K, and 100K tokens of unrelated interleaved images and text. This separates two effects. A tax at length zero is short-context degradation proper. Further decay with distance is the effective-context-gap phenomenon GIRAFFE reports.

Controlled extension and mechanistic probing. The third component is a controlled extension study. Published extended models conflate many choices, so we take one base vision-language model and extend it ourselves under matched compute. We vary three factors:

  • interpolation of all RoPE axes versus interpolation of the temporal axis only;
  • with versus without RoPE-perturbation training in the style of Shuffle the Context;
  • with versus without the 40% short-data mix.

On each variant we measure attention to the correct image region in the 30–70% layer band identified by CompoDistill. This tests whether the behavioral drop coincides with that attention becoming more diffuse.

Baselines

Naive reference: no extension. The base model, unmodified, scored on both static and procedural pairs. Every claim is a change relative to it. This rules out the trivial explanation that synthetic scenes are simply hard.

Simplest competing measurement: static benchmarks alone. This is the evaluation a skeptic would run instead of ours. It covers Winoground, ARO's VG-Relation and VG-Attribution, SugarCrepe, WhatsUp and MMVP-VLM, scored before and after extension on the same model pair. Our approach beats it only if the static drop measurably underestimates the procedural drop. If the two agree, the static audit wins and we say so.

Strongest existing short-context-preserving recipe: GIRAFFE. GIRAFFE combines M-RoPE++ with the 40% short-data mix. We evaluate the released GIRAFFE-Qwen2-7B against its Qwen2 vision-language base. We also reproduce the recipe inside our controlled study with identical data, token budget and learning-rate sweep, so that it competes on equal footing with the other variants.

For reference, the file reports these numbers:

  • GIRAFFE-Qwen2-7B: 65.9 on VideoMME and 61.3% on Visual Haystack.
  • GIRAFFE on Winoground, ARO or any minimal-pair test: no reported result found on Lacuna.
  • Long-CLIP on minimal pairs: no reported result found on Lacuna.
  • Winoground Group Score anchors: human 85.5%, chance 16.67%, VinVL 14.50% (Winoground).
  • SigLIP-B16 on Winoground: 10.3% Group Score raw and 72.5% with test-time matching (Test-Time Matching).

"Beating" GIRAFFE has a specific meaning here. A variant that leaves spatial axes unscaled would need to match GIRAFFE's VideoMME and Visual Haystack scores within noise while closing more of the procedural spatial-relation gap to the base model.

Contrastive pair: Long-CLIP. Long-CLIP versus CLIP gives a contrastive-model case on the same pairs, scored by image-text similarity.

Key-ingredient ablation: spatial interpolation. The ablation that isolates our proposed cause is spatial-axis interpolation on versus off, with everything else held fixed. If removing it does not recover spatial-relation accuracy, the position-encoding explanation fails. The alternative explanation would then be data-distribution forgetting, which the short-mix factor tests separately.

Why not just…?

Why not just run Winoground and ARO on extended models and stop there? This is the cheapest version and the real threat. It is also embedded in the design as the static arm. What it cannot do is distinguish "extension preserved binding" from "extension broke binding but memorized Visual Genome and COCO associations cover for it." ARO is built from exactly those crawled corpora, and DyVal shows that static scores can coexist with near-zero fresh performance. The static arm can still win: if its drops match the procedural drops on every relation type, the generator has added nothing for this purpose.

Why not just trust the 40% short-data mix to prevent forgetting? GIRAFFE's evidence for that mix comes from MME and MMBench, which are coarse recognition tests. The data mix also does nothing about geometric compression of the spatial position axes. The controlled study crosses the two factors directly. If the short mix alone restores procedural spatial accuracy, the problem is forgetting rather than positional geometry, and we would report that.

Why not just apply test-time matching, since compositional failure may be a scoring artifact? We report GroupMatch alongside GroupScore precisely to absorb this objection. Test-time matching itself is not a remedy for a model that has lost the binding signal. It amplifies existing confidence margins through repeated fine-tuning at inference. On rare or inverted triplets, those margins may come from priors rather than grounding. We will run test-time matching on base and extended models across the frequency knob. If it equally rescues extended models on rare triplets, the degradation is shallow and recoverable at inference, which would be a useful and publishable finding.

Why not just fine-tune with hard negatives in the style of NegCLIP after extension? NegCLIP's gains (VG-Relation from 63% to 81%) were measured on the same static sources it was trained on. Hard-negative training is reported to produce oversensitive perturbation detectors that reject valid paraphrases. We include it as a post-hoc repair arm and test it on procedural pairs with synonym-paraphrased positives. Whether its gains transfer to fresh scenes is open, and the experiment answers it.

Proposed Experiments

Experiment 1: the masking measurement. For each base/extended pair, we compute the extension drop on static benchmarks and on procedural pairs that match them in relation type. We test whether the procedural drop exceeds the static drop. Bootstrap confidence intervals are taken over items, and both GroupScore and GroupMatch are reported. The procedural set has several thousand verified pairs per relation type, regenerated with fresh seeds per evaluation so that it cannot leak.

Experiment 2: the frequency and remapping knobs. These test whether extension specifically hurts rare, inverted and remapped items. The test is an interaction effect in a regression of correctness on extension × triplet frequency × remapping. A positive interaction supports the claim that memorized priors are what survive extension.

Experiment 3: length sweep. We plot procedural accuracy against the amount of preceding distractor context for every variant. This separates the tax already present at zero length from decay with distance.

Experiment 4: the controlled ablation. This crosses spatial interpolation, perturbation training and short-data mix under matched compute. Long-context ability is checked on RULER, Visual Haystack and VideoMME, so that no variant buys short-context accuracy by abandoning long context. Middle-layer attention to the correct region is correlated with per-item correctness.

Decisive outcomes. The proposal succeeds if three things hold: procedural drops exceed static drops, the excess concentrates on spatial and rare relations, and turning off spatial interpolation recovers it without losing long-context scores. It fails cleanly if the static and procedural drops coincide.