When a model aligns a video with its narration, it needs a frame-by-frame matching. Methods that learn this matching end to end use a smoothed version of dynamic time warping, a dynamic program that finds the cheapest monotonic path through a matrix of frame-to-word distances. The distances come from separate video and text encoders. A separate literature shows that such encoders place the two modalities in different regions of the embedding space, offset from each other by a roughly constant vector. This is the "modality gap." We point out a simple algebraic fact. When that offset enters a cross-modal distance matrix, it adds terms that depend only on the row (the frame), only on the column (the word), or on neither. Inside a warping path, such terms do not measure correspondence at all. They change how many steps the path spends on each frame or word. In other words, the gap acts as a content-independent prior over durations, and an encoder trained through the alignment can learn to exploit it.
This matters for two reasons. First, the well-known failure modes of differentiable alignment are fought today with extra machinery: degenerate paths that collapse an action onto one frame, and rigid paths through irrelevant narration. Examples of that machinery are margin losses against shuffled transcripts and learned "dummy" tokens that absorb unmatched content. We hypothesize that part of what this machinery fights is the modality gap in disguise. Second, the modality-gap literature is split on whether the gap should be closed at all. We propose instead to make the alignment invariant to the gap while leaving the embedding space alone, which sidesteps that dispute. The method removes the additive part of each cross-modal cost matrix before warping. It then reintroduces genuine "this frame matches nothing" evidence through an explicit gate that is calibrated against modality-level statistics. The test is whether this beats the strongest weak-alignment method and also beats the obvious fix of simply centering the embeddings.
Core Claim
Write $x_i = \mu_x + a_i$ for the $i$-th video frame embedding and $y_j = \mu_y + b_j$ for the $j$-th text token embedding. Here $\mu_x, \mu_y$ are the modality means, and $a_i, b_j$ are the deviations from those means. The similarity then splits as
$$ x_i^\top y_j \;=\; \underbrace{a_i^\top b_j}_{\text{interaction}} \;+\; \underbrace{a_i^\top \mu_y}_{r_i} \;+\; \underbrace{\mu_x^\top b_j}_{c_j} \;+\; \underbrace{\mu_x^\top \mu_y}_{\kappa}. $$
For a monotonic warping path $\pi$, let $n_i(\pi)$ be the number of path cells in row $i$, let $m_j(\pi)$ be the number in column $j$, and let $|\pi|$ be the path length. The additive part then contributes $\sum_i n_i(\pi) r_i + \sum_j m_j(\pi) c_j + |\pi|\kappa$ to the path cost. Under soft-DTW with smoothing temperature $\gamma$, each path is weighted by $\exp(-\text{cost}/\gamma)$. So $\kappa/\gamma$ acts as a path-length prior. The terms $r_i/\gamma$ and $c_j/\gamma$ act as per-frame and per-word duration preferences: they favor lingering on frames that happen to project onto the text mean, regardless of content.
The claim has three parts. First, in pretrained and jointly trained video–text encoders, a measurable share of path mass is set by these additive terms. Second, end-to-end training grows that share, and its growth tracks path degeneracy. Third, an alignment that warps only on the interaction term, with relevance handled by a separate gate, improves step localization without closing the embedding gap. That last point is what keeps zero-shot retrieval intact.
Relation to Prior Work
The alignment side supplies the method. Soft-DTW replaces the hard minimum in dynamic time warping with a log-sum-exp, which gives gradients to every cell. OTAM uses the same relaxation for video-to-video few-shot matching. That setting is single-modality, so the additive terms cancel in expectation and the problem we study never arises. D3TW aligns video to discrete action labels and adds a hinge loss against permuted transcripts to stop collapse onto single frames. We read that collapse as partly a duration-prior pathology and test whether removing the additive terms reduces the need for the hinge. VT-TWINS is the closest work. It aligns HowTo100M clips and narrations with a locally smoothed soft-DTW, dummy elements for unmatched content, and an InfoNCE loss. Its costs are raw cross-modal inner products, so its dummy decision and its warping both see the gap-induced terms mixed together. Our proposal separates them. Perturbed DTW shows that soft-DTW equals expected minimum cost under Gumbel noise, and generalizes it to a nested variant (ns-DTW) that treats diagonal steps differently from horizontal and vertical steps. Horizontal and vertical steps are exactly where $r_i$ and $c_j$ act. Its results are on single-modality UCR series only, so we treat it as a compatible extension rather than a competitor.
The modality-gap side motivates the proposal and also constrains it. Mind the Gap shows the offset is universal across dual encoders, including VideoCLIP. It arises from a narrow initialization cone and is preserved by contrastive temperature. It also argues that the gap can help zero-shot accuracy and fairness, and Is the Modality Gap a Bug or a Feature? raises robustness concerns about closing it. Both are reasons not to close the gap in the embedding space. Closing the Modality Gap Aligns Group-Wise Semantics argues that ranking tasks are insensitive to the gap while distance-based tasks are sensitive to it. Dynamic programming over absolute costs is distance-based, so that argument motivates our diagnosis, though that paper works only on pooled vectors. I0T shows that much of the gap sits in a few peak dimensions and removes it by centering with batch means. Applied to tokens rather than pooled vectors, it is the most direct competitor to our method. CRESS shows the speech–text gap growing during decoding and regularizes output distributions. It addresses a different stage, generation rather than alignment, and we do not build on it.
Research Program
The first stage is a training-free diagnosis. We take frozen frame-level and step-text embeddings from CLIP-style and VideoCLIP-style encoders on CrossTask and Breakfast, using step or action names encoded as text for Breakfast. For each video–transcript pair we decompose the similarity matrix with a two-way additive model: grand mean plus row effects plus column effects, with the residual as the interaction. We report the fraction of variance in each part. We then run soft-DTW three ways: on the full matrix, on the additive part alone, and on the interaction part alone. Each resulting soft alignment is scored against ground-truth step boundaries.
If the additive-only alignment carries real but content-free structure, the diagnosis succeeds. "Content-free" means it is invariant to shuffling the transcript's text while keeping the transcript's length. "Real" means it is far from uniform. We also check how much of the row effect is explained by the projection onto the modality-mean direction $a_i^\top \mu_y$, as opposed to genuine frame irrelevance. Ground-truth background frames in CrossTask let us separate the two.
The second stage tests the shortcut hypothesis during training. We re-implement VT-TWINS-style training on a fixed HowTo100M subset with an S3D backbone. Throughout training we log the additive-variance fraction, the token-level centroid distance between modalities, and a degeneracy rate. The degeneracy rate is the fraction of transcript steps receiving one frame or fewer of soft path mass. The hypothesis predicts that additive share and degeneracy rise together, and that removing the dummy mechanism makes both rise faster.
The third stage is the method, which we call gap-invariant alignment. For each pair we double-center the cost matrix, subtracting row means and column means and adding back the grand mean, and run soft-DTW on the result. Because the additive part only affects visit counts, removing it leaves the warping path determined by content correspondence alone.
The removed row and column effects still carry useful information about whether a frame or word matches anything. They are therefore passed, after subtracting a running modality-level estimate of their expected value, to a small gate that decides how much mass goes to a dummy element. The running estimate is maintained the way I0T's learned normalization tracks modality bias, so it works on single samples at inference. This is the key design choice. Pair-level double-centering removes both nuisance offset and genuine relevance. The modality-level calibration returns only the relevance.
The embedding space is never centered, so pooled retrieval geometry is unchanged. The InfoNCE loss uses the resulting alignment score exactly as VT-TWINS does. A secondary variant swaps soft-DTW for ns-DTW inside the same construction. This tests whether direction-grouped transitions add anything once the duration prior is removed.
Baselines
The naive reference is MIL-NCE: temporal-agnostic contrastive learning that treats nearby clips as positives with no alignment. It is trained on the same HowTo100M subset, backbone, and steps. VT-TWINS reports that MIL-NCE reaches 35.5% average recall on CrossTask (VT-TWINS). This is the floor any alignment method must clear.
The strongest existing method is VT-TWINS itself, which reports 40.7% CrossTask average recall, 9.7% zero-shot YouCook2 text-to-video Recall@1, and 57.9% HMDB51 accuracy with S3D (VT-TWINS). Because we train on a subset, the published numbers are references only. The fair comparison is our re-implementation of VT-TWINS under the identical subset, compute, and a hyperparameter sweep of the same size. That sweep includes the smoothing temperature, the dummy threshold, and the neighborhood width. Beating it means a CrossTask recall gain larger than seed-to-seed variation across at least three seeds, with YouCook2 retrieval no worse.
For Breakfast alignment, D3TW reports 57.0% frame accuracy with ground-truth transcripts (D3TW). We rerun D3TW's hinge-loss objective with text-encoded action names in place of discrete labels, so that it becomes cross-modal. We compare with and without our double-centering. The published label-based number is cited as context, not as a matched comparison.
Three ablations isolate the ingredients. The first is VT-TWINS with I0T-style per-modality centering applied to token embeddings, which removes the offset in the embedding space rather than in the cost matrix. The second is pair-level double-centering without the calibrated gate, which tests whether relevance must be returned. The third is length-normalized soft-DTW, which divides by expected path length; it removes $\kappa$ but leaves $r_i$ and $c_j$ in place.
Why not just…?
Why not just center the token embeddings with I0T and run ordinary soft-DTW? This is the real threat. If the duration prior comes entirely from the global offset, token-level centering removes most of $r_i$ and $c_j$ at no cost. We see two reasons it may fall short, and we treat both as empirical questions. First, under end-to-end training the contrastive repulsion described in Mind the Gap can rebuild an offset that a fixed centering does not track. The second-stage logs will show whether it does. Second, centering moves the embedding space, and that space is what retrieval and zero-shot use. Mind the Gap's evidence that the gap can help zero-shot behavior suggests a cost there. If the centering ablation matches our method on CrossTask and YouCook2, we will report that the simpler fix suffices. We will then recast the contribution as the diagnosis plus that recommendation.
Why not just tune the temperature or normalize by path length? Length normalization removes only the constant $\kappa$. The per-frame and per-word terms set relative durations even at fixed path length, so normalization cannot remove them. Lowering $\gamma$ sharpens the alignment but scales the additive terms by the same $1/\gamma$ factor, so it cannot change their relative influence. The length-normalized ablation, run with the same temperature sweep, tests this directly.
Why not just close the gap with ModGap-style losses on tokens? Pulling true cross-modal pairs to zero distance at the token level presupposes the frame–word correspondence that alignment is supposed to discover. ModGap itself works only on pooled vectors. It also takes a side in the bug-versus-feature dispute that our method avoids. We include it as a secondary comparison only if the centering ablation is competitive, since it is the more invasive version of the same idea.
Why not rely on D3TW's hinge loss or VT-TWINS's dummy tokens, which already fight degeneracy? They may suffice. Our claim is that they partly compensate for a cause that can be removed directly. The test is whether double-centering lowers the degeneracy rate and reduces the gain those mechanisms provide. If adding our method on top of them changes nothing, the hypothesis is wrong.
Decisive Checks
The idea stands or falls on four observations. In the training-free diagnosis, the additive-only alignment must place non-trivial, transcript-shuffle-invariant mass on CrossTask videos; if it is near uniform, the duration-prior mechanism is negligible and the program stops there. During training, additive variance share must rise together with the degeneracy rate. On CrossTask, gap-invariant alignment must beat the matched VT-TWINS re-implementation and the I0T-centering ablation. On YouCook2, zero-shot retrieval must be preserved, because that is the point of making the alignment invariant to the gap rather than closing it.
References
- Soft-DTW: a Differentiable Loss Function for Time-Series
- Few-Shot Video Classification via Temporal Alignment
- D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation
- Video-Text Representation Learning via Differentiable Weak Temporal Alignment
- Perturbed Dynamic Time Warping: A Probabilistic Framework and Generalized Variants
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Is the Modality Gap a Bug or a Feature? A Robustness Perspective
- Closing the Modality Gap Aligns Group-Wise Semantics
- I0T: Embedding Standardization Method Towards Zero Modality Gap
- Understanding and Bridging the Modality Gap for Speech Translation