Tracking a Drifting Modality Gap Causally in Streaming Zero-Shot Captioning

Models such as CLAP (audio–text) and CLIP (image–text) are trained to place a sound or picture and its description at the same point in a shared embedding space. In practice they do not. Each modality occupies its own region, and the offset between the regions is called the modality gap. This offset is why a caption decoder trained only on text embeddings produces poor output when it is handed audio embeddings at test time. Every published fix treats the gap as one fixed displacement, measured on an offline dataset. In streaming audio the gap is not fixed. Background noise, reverberation and microphone changes move audio embeddings around over time, and a live system learns about each change only as it happens. It never sees it in advance.

We propose a lightweight module that sits between a frozen audio encoder and a frozen text-trained decoder, with no access to the future. We call it a relay, borrowing the communication-theory term for an intermediate node that re-encodes a signal before passing it on. Its job is to estimate the current drift from past embeddings and subtract a shrunken version of that estimate, but only inside the few embedding dimensions where the gap actually lives. The design question is borrowed from information theory's "dirty tape" channel, where a transmitter sees interference only as it arrives and linear partial compensation is the best Gaussian strategy. The empirical question is new: how much of the benefit of offline, whole-stream gap correction can a strictly causal module recover, and does restricting the estimate to the gap's subspace keep the module from subtracting away the content it is supposed to preserve?

Core Claim

Three bodies of work meet here, with unequal weight. The modality gap literature supplies the problem and the geometry. Causal state information supplies the operating constraint and a ladder of reference points. Relay transmission supplies the architecture: an uninformed decoder fed by an intermediate node.

On the problem side, Mind the Gap traces the gap to encoder initialization and contrastive repulsion. Training Audio Captioning Models without Audio and Diffusion Bridge show that the gap breaks text-trained decoders. Two geometry papers show the gap is concentrated in a small set of coordinates. The Double-Ellipsoid Geometry of CLIP separates image from text embeddings linearly with about 100% accuracy using 9 of 512 dimensions. I0T attributes the separation to modality-specific peak activations.

That concentration is the lever. If the offset lives in a low-dimensional subspace where caption content varies little, a causal estimator can track drift there without mistaking a long run of similar content (say, a minute of speech) for a shift in the gap. Our central claim is that a strictly causal, subspace-restricted, shrinkage-weighted linear relay recovers most of the captioning gain of an offline oracle that centers the whole stream at once. We further claim it beats the obvious causal alternative, a running mean subtracted in the full space, specifically on streams with long homogeneous content segments.

The relay performs the following operation:

$$\tilde z_t = \mathrm{norm}\big(z_t - \beta\, P\,(\hat s_t - \bar s)\big) - P\bar s .$$

Here $z_t$ is the audio embedding of window $t$ and $P$ projects onto a $k$-dimensional gap subspace, estimated once from paired audio–text data. The quantity $\bar s$ is the static offline gap between the audio and text means. The term $\hat s_t$ is an estimate of the current audio-side mean built only from $z_1,\dots,z_{t-1}$, or from $z_1,\dots,z_t$ in the causal-but-not-delayed variant. The gain $\beta\in[0,1]$ is a shrinkage factor, and $\mathrm{norm}$ is re-projection to the unit sphere.

The formal dirty-tape results of Achievable Rate Regions for Dirty Tape Channels assume the transmitter observes an additive Gaussian state that is independent of the message. Our state is instead estimated from observations that contain the message, so we do not import a capacity theorem. We import three things. The first is the linear form $X = U - \beta S$ with $\beta$ possibly below one. The second is the hierarchy of no state knowledge, strictly causal, causal and non-causal knowledge. The third is the sensitivity to one-step delay that this literature emphasizes. Each rung of the hierarchy becomes an experimental condition.

Shrinkage matters for a second reason. The gap is contested as a bug (Closing the Modality Gap Aligns Group-Wise Semantics) or a feature (Mind the Gap reports that a larger gap improves CIFAR10 zero-shot accuracy and reduces some FairFace offensive errors). Our relay corrects only the time-varying drift and leaves the static offset to a separately controlled term. That lets us ask whether the harm comes from the gap itself or from its instability.

Research Program

Constructing drifting streams. We first build streams whose drift we control. Clips from AudioCaps (AudioCaps) and Clotho are concatenated into long streams. Each stream is corrupted by a time-varying acoustic condition, meaning additive noise, reverberation and equalization, whose parameters follow a slowly varying autoregressive schedule. We vary two properties of each stream. The first is the drift timescale relative to clip length. The second is content homogeneity: clips are either shuffled randomly or grouped into long same-category runs. Because we generate the condition sequence, the true per-window drift is known. That allows us to compute a non-causal oracle and to measure state-estimation error directly.

The decoder and the offline gap. We use a caption decoder trained only on CLAP text embeddings, following the text-only recipe of Training Audio Captioning Models without Audio. We freeze it, along with the CLAP audio encoder. We estimate $P$ from clean paired data using a linear audio-versus-text classifier, following the separability analysis of The Double-Ellipsoid Geometry of CLIP. We also try the top modality-specific peak dimensions identified by I0T, and sweep $k$.

Estimating the drift. For $\hat s_t$ we compare an exponential moving average against a scalar Kalman filter per subspace coordinate. We choose $\beta$ in two ways: by the Kalman gain, or by validation on held-out streams.

Architectural comparison. The relay setting suggests a second question, drawn from Bounds on the Capacity of the Relay Channel With Noncausal State at the Source. That paper contrasts describing the state to an uninformed node with describing the input that node should use. The analogue here is to send the decoder either the corrected embedding, or a projection of it onto nearby text embeddings retrieved from a datastore, in the style of DRCap. We run this as a controlled comparison under the same causal drift estimate, so any difference is attributable to what is transmitted rather than to how drift is tracked.

Baselines

Uncorrected reference. The lower anchor is the frozen encoder feeding the text-trained decoder directly. It establishes how badly drift hurts.

Static correction. The second anchor subtracts the fixed offline offset $\bar s$ and nothing else. This is what all existing gap fixes effectively do in deployment. Beating it shows that the time-varying drift matters at all.

Running batch standardization. The strongest cheap competitor is I0T's training-free centering and renormalization. It reports Flickr30k image-to-text retrieval rising from 69.60% to 73.30% and text-to-image from 67.10% to 76.30%, with centroid distance falling from 0.7642 to 0.0102. We run it in a streaming form, with statistics computed over the past $W$ windows in the full embedding space, and sweep $W$ on the same validation streams used to tune our $\beta$ and $k$. We also run I0T's learnable batch-normalization variant with running statistics. Our method beats this baseline only if it wins at matched tuning budget, and specifically on homogeneous-content streams, where our hypothesis says full-space centering subtracts content.

Retrieval-augmented decoding. The strongest captioning system in the file is DRCap. It reports zero-shot Clotho METEOR of 18.2, matching supervised EnCLAP-large. We found no reported result on Lacuna for DRCap on AudioCaps with numbers. We reproduce DRCap on clean Clotho first, targeting METEOR near 18.2, before running it on drifted streams. Its datastore is static, so we test it with and without our relay in front.

Generative bridge. The strongest non-linear aligner is Diffusion Bridge, with MSCOCO CIDEr of 96.4 against 91.8 for CapDec. We retrain its one-dimensional UNet on CLAP text embeddings and report both caption quality and wall-clock latency per window.

Ablations. Four ablations isolate the key ingredients: full space versus gap subspace, $\beta=1$ versus shrunken $\beta$, strictly causal versus causal estimation, and the non-causal oracle that uses the whole stream's true drift. All methods share the same encoder, decoder, streams, and a hyperparameter search of equal size.

Why not just…?

Why not just subtract a running mean? This is the real threat. Our relay is a running mean with two restrictions added, so if full-space streaming I0T matches it, the honest conclusion is that simple causal centering suffices. The proposal's value would then reduce to measuring the causality penalty. We expect a difference only when content is locally homogeneous, because a full-space mean then absorbs content. The homogeneity sweep is designed to show that effect or refute it, and we will report either outcome.

Why not just use Diffusion Bridge or test-time inversion per window? Cross the Gap needs 150 to 1000 gradient steps per query, about 0.2 seconds per image, and Diffusion Bridge requires iterative sampling. Both also target a static gap learned from clean data. It is an empirical question whether they absorb drift anyway. We will measure it, along with latency, and consider them competitive only if they match our relay's quality within a real-time budget.

Why not just train the decoder to be robust, for example with noise injection? Noise-injected text-only training, as in the CapDec line that Diffusion Bridge compares against, may make the decoder tolerant of small shifts. We include a decoder trained with Gaussian noise injection at matched compute. If it closes the drift gap without a relay, the relay is unnecessary. We expect structured, drifting offsets to exceed the tolerance that isotropic noise buys, since I0T finds the gap is sparse and structured rather than isotropic.

Proposed Experiments

Captioning under drift. The primary metrics are CIDEr and METEOR on the drifted AudioCaps and Clotho streams, plus the same metrics on clean audio to verify that the relay does no harm. We summarize each method with a causality penalty:

$$\Delta = \frac{M_{\text{oracle}} - M_{\text{method}}}{M_{\text{oracle}} - M_{\text{none}}}.$$

Here $M$ is CIDEr under the non-causal oracle, the method, and no correction, respectively. $\Delta$ is the fraction of the oracle's recoverable gain that the method fails to capture. We report $\Delta$ against drift timescale, and against the difference between strictly causal and causal estimation, which the dirty-tape literature predicts should matter.

Decisive check: the homogeneity interaction. The single decisive result is the interaction between content homogeneity and estimator space. If subspace restriction lowers $\Delta$ relative to full-space streaming I0T on grouped streams, but not on shuffled streams, the confounding mechanism is confirmed. If there is no interaction, it is refuted.

Gap preservation. A second check addresses the bug-versus-feature question. With drift correction fixed, we vary the retained static offset and measure captioning, text-to-audio retrieval, and group-wise clustering V-measure. Our prediction is that caption quality depends mainly on drift stability, while retrieval tolerates a preserved static gap.

Generality. Finally, we repeat the core comparison on MSCOCO with photometric drift, using a CLIP text-only decoder. There we check whether our clean-stream CIDEr stays near Diffusion Bridge's 96.4 while our relay's latency per frame stays below theirs.