Some vision-language models mix image and text inside one network starting at the very first layer. This is called early fusion. Image pieces and words can attend to each other from layer zero, and this helps with spatial and compositional questions, such as which object is on the left or what changed between two images. Web-scraped training data is often misaligned, though. A dog photo captioned "my favorite Sunday afternoon" gives an early-fusion model no way to separate signal from noise, because the noise enters at the input and reaches every later layer. We propose that each image-text pair should decide how early it gets fused. A small, cheap estimate of how well the caption matches the image scales down cross-modal attention in the early layers, and it also down-weights the cross-modal masked-prediction loss. Poorly matched pairs are not discarded. They fall back toward late fusion, where image and text are processed separately and only compared at the end, so they still teach unimodal and contrastive structure.
The core claim is testable. Web noise harms early fusion more than it harms late fusion. Gating the depth of fusion by per-pair alignment should keep early fusion's reasoning advantage at noise levels where ungated early fusion loses it. It should also do better than the obvious fix of filtering the data first. If filtering matches the gated model at equal compute, the claim is false, and the experiments are designed so that outcome is visible.
Core Claim and Motivation
The motivating evidence comes from three places. FuseLIP quantizes each image into 128 discrete tokens and concatenates them with text tokens at layer zero. It trains with a sigmoid contrastive loss plus masked multimodal modeling, which predicts masked image or text tokens from both modalities. FuseLIP reports 94.2% on CC12M-TGIT versus 81.0% for a late-fusion SigLIP variant, and 68.9% on OI-Pos versus roughly 48%. It has no mechanism for noisy pairs, and its masked-prediction objective explicitly forces image tokens to explain caption tokens even when the caption is about something off-frame.
Two probing studies explain why that matters. Compose and Fuse finds that early layers of multimodal models identify which modality a token comes from but cannot separate task-relevant facts from distractors. In its Complementary setting, accuracy falls from 94.6% to 52.0%. Raising the softmax temperature of early-layer attention at inference partly recovers this. Where Does Vision Meet Language? shows that fusion concentrates in layers 0–4, and that high attention on irrelevant regions in those layers persists downstream. It applies a training-free contrastive-attention mask that lifts the LLaVA-1.5 six-benchmark average from 55.19 to 58.25. Both papers motivate our proposal, and both intervene only at inference. We move the intervention into training, and we make it depend on the data pair rather than applying it uniformly.
The method draws on the overfitting literature. CM-VQVAE learns masks that prune "contradictory" features after early concatenation. Multimodal Information Bottleneck compresses a joint early-fused representation. Both show that unconstrained early fusion memorizes spurious cross-modal correlations, but both were tested only on small sentiment and emotion classifiers. We borrow their premise that cross-modal capacity should be spent selectively. We implement it as an attention gate inside a transformer, not as a feature-vector mask or a Gaussian bottleneck.
Mechanism
Let $x = (v, t)$ be an image-token sequence $v$ and a caption $t$. A lightweight alignment head computes $g(x) \in (0, 1]$ from pooled layer-0 embeddings of each modality, before any cross-modal mixing. Because it sees no cross-modal mixing, this head is effectively a tiny dual encoder. Its targets are soft. An exponential-moving-average copy of the model scores each pair against the other pairs in the batch, and those scores supervise the head. This is the momentum-distillation idea that ALBEF used for late-fusion encoders; the file describes ALBEF but has no separate entry for it.
In layers $\ell < L$, every attention logit from a token of one modality to a token of the other is shifted:
$$s'_{qk} = s_{qk} + \log g(x) \quad \text{for } q, k \text{ in different modalities},\ \ell < L$$
Here $s_{qk}$ is the usual scaled dot-product logit. Layers at or above $L$ fuse without restriction. When $g$ is near 1, the model behaves like FuseLIP. When $g$ is small, it behaves like a model that fuses late.
The cross-modal part of the masked-prediction loss is weighted by $g(x)$ as well, so that the model is not rewarded for "explaining" a caption with unrelated pixels. The gate is stopped-gradient with respect to the reconstruction loss. This prevents the model from shrinking $g$ simply to make prediction easier.
Research Program
The study has three parts.
The first part measures the problem. We pretrain FuseLIP-style early-fusion models and SigLIP-style late-fusion models on CC3M and CC12M. Clean subsets are corrupted with controlled caption shuffling at 0%, 20%, 40% and 60%. We also keep the natural noise and stratify it by a Sieve-style score, which compares a captioner's description with the alt-text. The test is whether early fusion's margin on OI-Pos and TGIT shrinks faster with noise than late fusion's margin. If it does not, the premise fails and the gate is unnecessary.
The second part trains the gated model under the same settings and examines what the gate learns. We check whether $g$ correlates with injected corruption, whether low-$g$ pairs are disproportionately "non-literal but informative" captions, and how early-layer attention entropy changes. The third part addresses overfitting. We fine-tune each pretrained model on small seed sets of spatial and grounding tasks (OI-Pos, VG-Crop, CC3M-VQA) at several training-set sizes, and measure the gap between training and held-out accuracy.
Baselines
The naive reference is ungated FuseLIP pretrained on the raw, noisy corpus. This is the "do nothing" point that every intervention must beat, and on clean data it reproduces the reported 94.2% TGIT and 68.9% OI-Pos (FuseLIP). The late-fusion SigLIP variant from the same paper is the second reference, at the reported 81.0% TGIT and about 48% OI-Pos. Both are retrained by us on each noise condition with identical token budgets and identical hyperparameter sweeps.
The strongest competitor on this question is filtering followed by ungated early fusion. We prune the corpus with a Sieve-style captioner-versus-alt-text distance, or with BLIP CapFilt-style caption replacement, and then train FuseLIP. It is the strongest because it attacks the same cause with a mature tool. No reported result was found on Lacuna for Sieve- or CapFilt-filtered data in an early-fusion setting. Comparison is fair only at equal training steps, so the filtered model sees its smaller set for more epochs. Filtering thresholds get the same tuning budget as our gate hyperparameters ($L$ and the gate temperature). Our method beats this baseline if it is better on OI-Pos and TGIT at 40% or more injected noise by more than seed-to-seed variance across three seeds, and is not worse on clean data.
The inference-time competitor is early-layer attention softening as in Compose and Fuse, applied with a uniform temperature to the ungated noisy-trained model. The file reports no number for this intervention on FuseLIP-type tasks, so we tune its temperature on the same validation split.
Three ablations isolate the key ingredient, which is the per-pair alignment signal. A constant gate tuned to the best single value tests uniform delayed fusion. A random gate with matched mean tests whether any stochastic regularization helps. Gating attention only, or gating the masked-prediction loss only, shows which of the two pathways carries the effect.
Why not just…?
Why not just filter the data and keep plain early fusion? This is the real threat. If noise is mostly whole-pair garbage, filtering removes it cheaply and our gate adds nothing. Our bet is that much web noise is partial: the caption is relevant to the image but non-literal. The file notes that strict pruning discards metaphor and world knowledge. Experiment one settles it. If filtered FuseLIP matches the gated model on natural CC12M noise, we report that, and we narrow the claim to synthetic or extreme noise. A combined filter-plus-gate run tests whether the two are complementary.
Why not just fuse later for every pair? A fixed fusion depth is the constant-gate ablation. FuseLIP's spatial gains depend on interaction at layer zero, and Contrastive Attention locates most fusion in layers 0–4. Delaying fusion uniformly should therefore cost reasoning accuracy on clean pairs. If the constant gate ties with the learned gate, the per-pair signal is not doing the work, and we would say so.
Why not just soften attention at inference, as Compose and Fuse does? Inference-time softening cannot undo associations already burned into the weights by masked prediction over misaligned pairs. It also treats a clean pair and a junk pair identically. The direct comparison is the tuned-temperature baseline above, applied on top of noisy-trained weights.
Proposed Experiments and Decisive Checks
All models use the FuseLIP-B architecture and TiTok tokenizer, trained on CC3M and CC12M under the noise schedule above. Evaluation uses CC12M-TGIT, OI-Pos, VG-Crop, OI-Crop and CC3M-VQA for compositional and spatial reasoning, plus ImageNet zero-shot as a check that unimodal ability survives. As a transfer probe of the early-bias diagnosis, we rebuild the Compose and Fuse Complementary pattern as a retrieval task, where the facts needed are split across image and text. We ask whether gated pretraining narrows the gap that paper reports between 52.0% and 94.6%. That figure comes from MLLMs, so we compare only relative gaps within our own models.
The first decisive check is the interaction plot of fusion type against noise rate. Ungated early fusion should lose more of its advantage under noise than late fusion does. The second is the gated model against filtered FuseLIP at 40% and 60% noise and on natural noise. The third is gate validity: the area under the curve of $g$ for detecting injected shuffles, plus a human audit of 500 low-$g$ natural pairs to check whether they are junk or non-literal. The fourth is the small-sample fine-tuning curve. If gating limits spurious cross-modal capacity in the way CM-VQVAE argues masking does, the gap between training and held-out accuracy on 1–10% seed sets should be smaller.
A negative result on the second check, combined with a positive result on the first, would still be useful. It would establish that early fusion is noise-fragile and that curation, not architecture, is the right fix.
References
- FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens
- Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning
- Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention
- Complementary-Contradictory Feature Regularization Against Multimodal Overfitting
- Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations
- SIEVE: Multimodal Dataset Pruning Using Image Captioning Models
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation