Two-Layer Relay Codes for the Modality Gap

Models such as CLIP place images and text in one shared vector space, yet the two modalities occupy separate regions of it. This "modality gap" does not hurt retrieval, because retrieval only ranks candidates. It does hurt tasks that need absolute positions to agree: clustering, or decoding an image vector with a text-only decoder. The literature disagrees on what to do about it. One line argues the gap should be driven to zero. Another shows that a non-zero gap improves zero-shot accuracy and fairness. We propose not to pick one gap. A small module sits between frozen encoders and a frozen decoder and emits a two-layer code. The coarse layer is decoded by tasks that only need matching, and it keeps the original geometry. The fine layer is decoded only by tasks that need reconstruction, and it carries just enough extra information to move a vector onto the other modality's manifold.

The design comes from network information theory, the study of codes for networks with several senders, receivers, and helper nodes. It borrows two specific results. The first is the source-coding result of Katz et al.. They encode a source with a coarse description $U$ that suffices for detection and a refinement $V$ that suffices for estimation, in a nested chain $U - V - X - Y$. They show that when the test is against independence, the rates for the two tasks separate cleanly. The second is the relay idea. A helper node between sender and receiver forwards a compressed index and exploits side information instead of fully decoding, as in Gattegno et al. and the lookahead relays of Wang et al.. In our setting the adapter is the relay. The target modality's own unpaired data is side information available at the decoder.

Core Claim

Retrieval and cross-modal reconstruction behave like detection and estimation in the Katz chain. If that holds, one fixed gap-closing map is the wrong object. A single map must commit to one geometry, and Liang et al. show that this geometry is task-dependent: tuning a non-zero gap raised CIFAR10 zero-shot top-1 from 0.9013 to 0.9081 and cut FairFace denigration for one group from 15.7% to 13.7%. Meanwhile Grassucci et al. need near-zero centroid distance to roughly double MSCOCO clustering V-Measure, from about 13.0 to 23.63.

A two-layer code can serve both regimes:

  • The coarse layer $U$ is the untouched frozen CLIP embedding, so retrieval and zero-shot behavior are preserved by construction.
  • The fine layer $V$ is a low-rate refinement. The decoder combines $V$ with side information $Y$ retrieved from an unpaired bank of target-modality embeddings, using $U$ as the query.

The claim makes two testable predictions. First, $V$ needs far less capacity than a full image-to-text regressor, because $Y$ already supplies most of the target manifold's shape. Second, strengthening the retrieval layer should leave reconstruction quality essentially unchanged, mirroring the Katz separability result.

Research Program

The source is a frozen image encoder (and audio, for a trimodal check). The destination is a frozen text-embedding consumer. The prime example is a captioning decoder trained only on text embeddings, as in the zero-shot captioning setting of Lee et al.. The relay is a small network that sees the image embedding $x$ and outputs $V$. At decoding time, $U = x$ is used to retrieve the $k$ nearest embeddings from a bank of unpaired text embeddings; this is $Y$. A light decoder $g(U, V, Y)$ outputs a point on the text manifold.

Training is a variational information bottleneck, meaning a training objective that charges a cost for every bit a latent variable carries. Here it charges only the refinement layer:

$$\min \; \mathbb{E}\, \| g(U,V,Y) - t \|^2 \; + \; \beta\, I(V; X \mid U, Y),$$

where $t$ is the paired text embedding (only a small paired set is used) and $\beta$ sets the refinement rate. The conditional rate $I(V;X\mid U,Y)$ is the Katz refinement term. It is estimated with a Kullback–Leibler bound against a prior conditioned on $(U, Y)$. Because the retrieval layer is fixed, its "detection exponent" is simply CLIP's own retrieval quality.

As an extension, we let $U$ be lightly fine-tuned with a contrastive weight $\lambda$. Sweeping $\lambda$ against $\beta$ then traces an empirical rate–retrieval–distortion surface. This surface is the learned analogue of the $(R, E, D)$ region that Katz et al. compute for binary sources.

The relay framing does real work in one place. Cooperative binning in Gattegno et al. forwards a bin index rather than the payload. Our variant makes $V$ a discrete index of small cardinality, chosen so that it disambiguates among the $k$ retrieved neighbors. This lets us measure in bits how much image-specific information is actually needed to cross the gap. That quantity is new: existing gap work reports distances, not rates.

Baselines

The naive reference is doing nothing: frozen CLIP image embeddings fed directly to the text-only decoder. A second naive reference subtracts the centroid offset between the two modalities. Both bound how much of the problem a constant shift explains.

The strongest single-pass competitor is a direct regression adapter: an MLP of matched parameter count mapping image embeddings to paired text embeddings. It uses the same paired subset, the same unpaired bank (supplied as extra input so the comparison is fair), and the same tuning budget of hyperparameter trials. This is the baseline the proposal must beat on its central claim.

The strongest existing method for captioning across the gap is Diffusion Bridge. Lee et al. report 96.4 CIDEr on MSCOCO zero-shot captioning, against 93.3 for C3 and 91.8 for CapDec. We would rerun it with the same decoder and text corpus. Beating it means matching its CIDEr within about one point at a small fraction of its inference latency, or exceeding it outright.

For clustering, the reference is ModGap's 23.63 MSCOCO V-Measure and 32.1 on MSR-VTT (Grassucci et al.). For retrieval, CS-Aligner reports 88.7 average R@1 on Flickr30k (Yin et al.). CS-Aligner retrains the alignment, so our goal on retrieval is not to beat it but to show the coarse layer loses nothing relative to frozen CLIP.

Two ablations isolate the key ingredients. The first removes the side information $Y$, so $V$ must carry everything. The second collapses $U$ and $V$ into one latent with one rate penalty. If either ablation matches the full model, the corresponding ingredient is not doing the work.

Why not just…?

Why not just train a small MLP that maps image embeddings onto text embeddings? This is the real threat. It is single-pass and fast, and Lee et al. only assert, rather than show on matched budgets, that linear projections lack precision. The MLP's weakness is structural: it outputs one geometry. If downstream tasks reuse its output, the CIFAR10 and FairFace effects reported by Liang et al. predict a cost on zero-shot accuracy and fairness. Our experiments evaluate the MLP's output on every task, not just captioning. If the MLP matches us on CIDEr and V-Measure without hurting zero-shot accuracy or FairFace, the hierarchy is unnecessary and we will report that. The residual contribution would then be the rate measurement alone.

Why not just retrain CLIP with a gap-closing loss such as ModGap or CS-Aligner? These methods require retraining the encoders and tuning sensitive weights. Grassucci et al. report weaker gains under heavy label imbalance such as CIFAR10. More fundamentally, they impose one gap on every consumer. We compare against their reported numbers on MSCOCO and MSR-VTT clustering. We also check whether our frozen coarse layer avoids the zero-shot regressions a uniformly closed gap would cause.

Why not just subtract the mean offset? Levi et al. describe the space as two off-center ellipsoidal shells in narrow cones, which a translation cannot reshape. The do-nothing and mean-shift baselines measure directly how much of the captioning and clustering deficit a shift recovers.

Proposed Experiments

The main experiment runs MSCOCO zero-shot captioning (CIDEr) and Flickr30k captioning. It also measures MSCOCO and MSR-VTT clustering (V-Measure, centroid distance), Flickr30k retrieval R@1, CIFAR10 and Caltech101 zero-shot accuracy, and FairFace denigration rates. For reference on Caltech101, Fahim et al. report 0.42 for default CLIP and 0.49 with their constraints. All methods sit on the same frozen backbone, and wall-clock latency per sample is reported throughout.

Three checks decide the claim:

  1. Rate. Sweep $\beta$ and plot CIDEr against the measured rate of $V$. A knee at a few bits per sample would show that crossing the gap needs little image-specific information once side information is present.
  2. Separability. Sweep $\lambda$ on the coarse layer at fixed $\beta$. Flat captioning quality would support the Katz-style separation of retrieval from reconstruction. A clear coupling would mean the regime resembles the open general-hypothesis case, which is itself informative.
  3. Single map versus two layers. The matched MLP and the single-latent ablation are scored on all tasks at once. The proposal stands only if the two-layer code is the only system that keeps frozen-CLIP zero-shot and fairness numbers while also reaching Diffusion-Bridge-level captioning and ModGap-level clustering.