Removing CLIP's Modality Signature in One Step, Band by Band

Vision-language models such as CLIP place images and captions in one shared vector space, but the two kinds of vectors never fully mix. Image vectors and text vectors sit in two separate clusters, offset from each other by a consistent displacement called the modality gap. The result is that a caption generator trained only on text vectors hallucinates when it receives an image vector. Diffusion Bridge fixes this by training a diffusion model on text vectors. A diffusion model is a generator that learns to turn noise back into data by removing it step by step. Diffusion Bridge treats the image-to-text offset as if it were noise, adds noise to the image vector, and then runs many denoising steps to pull it onto the text cluster. That raises paired cosine similarity from 0.306 to 0.606 and reaches a zero-shot MS-COCO captioning CIDEr of 96.4. The authors themselves flag the multi-step sampling as a latency bottleneck.

This note argues that Diffusion Bridge is slow and lossy for the same reason. It adds noise equally in every direction of the space, although the gap lives in only a few directions. Levi and Gilboa show that 9 of CLIP's 512 dimensions separate images from text almost perfectly. The analogy is transform-domain watermarking, where designers change only the frequency coefficients in which a change is both effective and harmless. Here the modality gap is a kind of signature that CLIP stamps on every image vector, concentrated in a few spectral coefficients. We propose to erase it band-selectively. Heavy noise goes into the coefficients that carry the signature, and little goes into the coefficients that carry image content. A one-step average-velocity flow model (MeanFlow) then jumps to the text cluster in a single network evaluation. The claim is that this schedule makes the text-ward trajectory nearly straight, so one step suffices, and that it preserves more image content than uniform noising.

Core Claim

We express embeddings in a spectral basis: eigenvectors of the pooled embedding covariance, ranked by how strongly each direction separates image from text statistics (a Fisher ratio). The top-ranked "gap band" corresponds to the few separating directions reported by Levi and Gilboa. The rest form the "content band." Let $x$ be a clean text vector and $\epsilon$ Gaussian noise. We use a single path parameter $s \in [0,1]$ and give each band $b$ its own noise level $\tau_b(s)$:

$$x_s^{(b)} = (1-\tau_b(s))\,x^{(b)} + \tau_b(s)\,\epsilon^{(b)}, \qquad \tau_{\text{gap}}(s)=s,\quad \tau_{\text{content}}(s)=\alpha s,$$

Here $\alpha \in [0,1]$ controls how much content-band noise is used. Setting $\alpha=1$ recovers uniform, Diffusion-Bridge-style noising. Because the path is still indexed by one scalar $s$, MeanFlow's average-velocity identity applies unchanged. Let $u(x_s,s,r)$ be the average velocity from time $s$ to time $r$, and $v$ the instantaneous velocity of the path. MeanFlow's identity is

$$u = v - (s-r)\,\frac{du}{ds}.$$

This lets us train a single-step jump from the injection level $s=M$ to $r=0$ on text vectors alone, with no pretrained teacher.

At inference, an image vector is noised according to the schedule at level $M$ and mapped in one evaluation. The two predictions are:

  1. Some $\alpha<1$ beats $\alpha=1$ at equal one-step cost.
  2. The learned trajectories are measurably straighter at that $\alpha$.

The mechanism mirrors what FARI found for image diffusion inversion. There, trajectories anchored by preserved coarse structure have low curvature and compress to one step. In our setting, the content bands barely move, so curvature should be confined to the small gap band.

Relation to Prior Work

Diffusion Bridge supplies the premise: the gap can be treated as noise and removed by denoising. We keep its text-only training regime and its GPT-2 caption decoder. We change two things: noise is applied band-selectively, and sampling takes one step. MeanFlow supplies the one-step training method. It has been shown only for image synthesis, with a 1-NFE FID of 3.43 on ImageNet 256×256, and never as a bridge between embedding distributions. FARI motivates the curvature argument, but it concerns image-latent inversion and involves no cross-modal geometry.

Three papers constrain the design:

  • FSAlign shows that global center-shifting distorts local neighborhood rankings. This is why we do not simply delete the gap band.
  • Levi and Gilboa show that the gap has curved, ellipsoidal structure, with vector norm encoding "conformity" (an instance's average similarity to all others). This is why a single constant offset is unlikely to suffice.
  • Grassucci et al. show that retrieval depends only on rankings and is therefore insensitive to the gap, whereas clustering and decoding depend on distances and are sensitive to it. We therefore evaluate on decoding, where the gap matters, and use retrieval only as a check that content survives.

On frequency-selective embedding, the watermarking lineage supplies the design principle rather than the application. That lineage includes mid-band DCT embedding (Amin et al.), content-adaptive band choice (Mao et al.), and null-space placement in Shallow Diffuse. The shared principle is to choose coefficients by where a perturbation has effect and where it does harm. We use this principle to remove a signature rather than insert one. None of these papers touches multimodal latents or one-step bridging.

Research Program

Spectral anatomy of the gap. The first stage is a measurement study on frozen CLIP ViT-B/32 using MS-COCO embeddings. We decompose the image-text offset in the eigenbasis and ask three questions. How concentrated is the gap's energy, and does it match the 9 separating dimensions? How much caption-relevant content lives in the top-Fisher directions? We measure this as the drop in text-to-image retrieval when those directions are replaced by noise. Finally, how does curvature distribute across bands along Diffusion Bridge's own multi-step trajectory? As a secondary basis, we will also use eigenvectors of a nearest-neighbor graph Laplacian on the text cloud, which are true manifold "frequencies" related to FSAlign's heat kernels. This tests whether the conclusion depends on the choice of basis.

Band-selective one-step bridge. The second stage trains MeanFlow average-velocity models on text embeddings under the anisotropic schedule above. We sweep $\alpha$, the injection level $M$, and the gap-band size. Band identification uses unpaired image statistics only, the same information a mean-offset baseline needs, so no image-caption pairs enter training. We report caption quality, paired cosine similarity, and per-embedding latency.

Baselines

Naive reference points. The first is to feed raw image vectors to the text-trained decoder, which measures the damage the gap does. The second subtracts the image-text mean offset before decoding. This is the zero-cost fix a reviewer would try first. Diffusion Bridge reports that linear mappings and simple noise injection hallucinate. We re-run both under our decoder to have our own numbers.

Strongest existing method. This is Diffusion Bridge itself. We reimplement it with the same text corpus, the same CLIP backbone, the same GPT-2 decoder, and an equal hyperparameter-tuning budget. We run it at its full step count and also report wall-clock time. The reported number to beat is CIDEr 96.4 on the MSCOCO Karpathy split, against C3 at 93.3 and CapDec at 91.8. I found no reported latency or step-count figure on Lacuna for Diffusion Bridge, so we measure it ourselves. Beating it means matching its CIDEr within about one point in a single network evaluation, or exceeding it at equal or lower cost.

The cheap version of the strong method. We run Diffusion Bridge with 1, 2, and 4 deterministic DDIM steps, where DDIM is the standard sampler that skips steps along a deterministic path. If the uniform-noise trajectory is already straight, this closes most of the gap for free.

Isolating the key ingredient. The critical ablation is an isotropic MeanFlow bridge ($\alpha=1$). It is trained identically, differing only in the schedule, so any gain of the full method over it comes from band selectivity alone. A second ablation assigns heavy noise to randomly chosen bands of the same size. This separates "anisotropy helps" from "the gap bands specifically matter."

Why not just…?

Why not just subtract the mean offset? The gap is not a constant vector. Levi and Gilboa show two off-center ellipsoidal shells whose norms encode conformity, and FSAlign shows that global shifting damages local neighborhoods. Still, this is cheap, and the anatomy study will report exactly how much caption quality it recovers. If mean subtraction plus the gap band alone recovers most of the CIDEr, that is itself a useful and simpler finding, and we would report it as such.

Why not just run Diffusion Bridge with two DDIM steps? This is the most serious threat. If uniform-noise trajectories are already nearly straight, few-step DDIM would erase the latency argument. The anatomy study measures that curvature directly. The CIDEr-versus-steps curve settles the question. If 2-step DDIM comes within one CIDEr point of the full sampler, the contribution narrows to the content-preservation claim, which must then stand on the $\alpha$ sweep alone.

Why not just distill Diffusion Bridge into one step? Progressive distillation (Huang et al.) could yield a one-step student. However, it inherits the teacher's uniform noising, and therefore the teacher's trade-off between removing the gap and destroying content. We include a distilled uniform-schedule student alongside the isotropic MeanFlow ablation. If both match the band-selective model, anisotropy is not the operative ingredient.

Why not just regress image vectors onto text vectors with an MLP? Regression needs paired data, which breaks the text-only zero-shot setting of Diffusion Bridge and CapDec. It also predicts a conditional mean that may fall off the text cluster. We report it as a paired upper-reference, not as a like-for-like competitor.

Proposed Experiments

Primary benchmarks. The primary benchmark is MSCOCO Karpathy-split zero-shot captioning, measured by CIDEr, with Flickr30k as a second split. I found no reported Flickr30k number on Lacuna for Diffusion Bridge. We also report paired image-text cosine similarity against the reported 0.606, and latency per embedding in network evaluations and wall-clock time.

Content-preservation checks. We run MS-COCO image-to-text and text-to-image retrieval using the bridged vectors. For reference, FSAlign reports 51.26% and 31.20% Recall@1 on ViT-B/32. Retrieval is included only to verify that content bands survive, since Grassucci et al. show that retrieval is gap-insensitive. We also measure clustering V-Measure, where ModGap reports 23.63 on MSCOCO, to test whether bridged image vectors group by caption semantics.

Decisive checks. Three results decide the claim:

  1. The $\alpha$ curve. At one step, CIDEr should peak at some $\alpha<1$ above the isotropic $\alpha=1$ model. A flat curve or a peak at 1 refutes the band-selectivity claim.
  2. Straightness. Measured trajectory curvature at the best $\alpha$ should be lower than at $\alpha=1$, confirming the FARI-style mechanism rather than an incidental regularization effect.
  3. Random-band control. Replacing the Fisher-ranked gap band with random bands of equal size should erase the gain.

We also transfer the method to a second backbone, OpenCLIP or SigLIP (both evaluated by FSAlign), to test whether gap bands found on one model's statistics generalize. If all three decisive checks pass and one-step CIDEr is within a point of full Diffusion Bridge, the result is a real-time modality bridge with a clear explanation for why a single step suffices.