When one search engine must rank text pages, images, and image-plus-text documents in a single result list, image-text models such as CLIP fail in a specific way. Text queries push text candidates above equally relevant images, because each modality's embeddings sit in their own region of the shared space. That systematic separation is called the modality gap. The fixes proposed so far change the embedding geometry. They either subtract each modality's average vector, or retrain the encoders until the regions merge. This proposal starts from a different observation. A mixed list is really two decisions: how to order candidates within each modality, and how to interleave the modalities. The gap damages only the second.
We propose to measure that decomposition directly on a mixed-modality benchmark and then repair only the interleaving. The repair is a small, query-conditioned calibration head trained with a listwise ranking loss over mixed lists, with the encoders frozen. If the hypothesis holds, the work does two things. It explains why retrieval inside a single modality is insensitive to the gap while mixed search is not. It also eases the field's disagreement over whether the gap should be closed or kept: the embedding geometry, and whatever benefits it carries, can be left alone while the ranker is corrected.
Core Claim
The listwise model we use is the Plackett–Luce model, a probability distribution over orderings in which each next item is chosen with probability proportional to the exponential of its score. Under this model, the ordering it implies over any subset of candidates depends only on those candidates' scores, and adding the same constant to all of them changes nothing. The relevant classical choice models are unified in Fast and Stable Maximum Likelihood Estimation for Incomplete Multinomial Models, which places Bradley–Terry (pairwise) and Plackett–Luce (listwise) under one maximum-likelihood framework.
Write $s(q,c)$ for the score of candidate $c$ under query $q$. Then any score distortion that acts on a whole modality pool as a per-query offset leaves every within-modality ordering untouched. It moves probability only across modality boundaries. This is the ranking-side counterpart of the result in Closing the Modality Gap Aligns Group-Wise Semantics, which proves that single-pool retrieval is invariant to a uniform modality shift.
Our claim has two parts. The first is that most of the ranking loss CLIP suffers on mixed-modality search is interleaving loss. The second is that the part mean subtraction leaves behind is a mismatch in scale rather than location: text-to-text similarities and text-to-image similarities have different spreads, and that difference varies with the query. Both residual problems can be corrected by the score model
$$ \tilde s(q,c) = a_{m(c)}(q)\,\cos(q,c) + b_{m(c)}(q), $$
where $m(c)$ is the candidate's modality type, with fused image-plus-text documents treated as a third type. The functions $a_m$ and $b_m$ are a small network reading the frozen query embedding and producing a scale and an offset for each modality type.
The model is trained by maximizing the Plackett–Luce likelihood of relevance-sorted mixed lists:
$$ \mathcal{L} = -\sum_{i=1}^{n} \log \frac{\exp \tilde s(q,c_{\pi(i)})}{\sum_{j \ge i} \exp \tilde s(q,c_{\pi(j)})}, $$
where $\pi$ is a ground-truth ordering of the $n$ candidates. Because the offset $b_m$ cancels inside each pool, this objective spends its capacity entirely on interleaving. Within-pool order can still change through the scale $a_m$ only if $a_m(q)$ is negative, which we constrain away.
Relation to Prior Work
The motivating evidence comes from Closing the Modality Gap for Mixed Modality Search. That paper defines mixed-modality search, introduces the MixBench benchmark built on OVEN and VisualNews, names intra-modal ranking bias as the failure, and fixes it with GR-CLIP. GR-CLIP subtracts the modality means estimated on a small calibration set and then scores each candidate independently by inner product. Our proposal differs in three ways:
- GR-CLIP corrects location only, while we also correct scale.
- GR-CLIP's correction is the same for every query, while ours is conditioned on the query.
- GR-CLIP's correction is unlearned, while ours is fit with a ranking objective defined over mixed lists rather than on the embeddings.
Note that mean subtraction already induces a query-dependent score shift of $-q\cdot\mu_m$. A learned global offset therefore does not subsume it, and the hierarchy of baselines below is built with this in mind. I0T: Embedding Standardization Method Towards Zero Modality Gap is a sibling post-hoc method. It standardizes embeddings per modality and finds modality-specific activation spikes, which suggests that per-modality variance, and not only the mean, differs. That observation motivates our scale term.
Two papers constrain the design and frame the debate about whether to close the gap. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning explains the gap through cone-shaped embeddings produced at initialization and through contrastive training dynamics. It shows that enlarging the gap can improve zero-shot accuracy and reduce offensive misclassifications on FairFace. ModGap, from Closing the Modality Gap Aligns Group-Wise Semantics, retrains the encoders to remove the gap for clustering. Neither paper studies mixed-list ranking. Our design deliberately leaves the embeddings unchanged, so it cannot cost the benefits Liang et al. report, and we test whether it matches the gain from closing the gap on ranking.
The ranking-side methods come from two lines of work. Integrating Listwise Ranking into Pairwise-based Image-Text Retrieval adds a smooth NDCG loss to dual encoders. It derives graded relevance from a text sentence encoder applied to captions, and it ranks only single-modality pools, so it never confronts interleaving. Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks shows that pointwise, pairwise, and listwise scorers are special cases of a network that scores a group of $m$ items at once. It has been tested only on single-modality learning-to-rank data, and it reports that gains plateau beyond small group sizes. We use it as the expressive upper rung of our hierarchy, to test whether interactions between candidates add anything beyond calibration.
Research Program
The first stage is diagnostic. For each system on MixBench, we compute NDCG@10 (normalized discounted cumulative gain over the top ten results) for the system's own list and for two oracle hybrids:
- An oracle interleave, which keeps the system's within-modality orderings but merges the modality pools according to ground-truth relevance.
- An oracle within-order, which keeps the system's modality pattern for each rank position but fills each slot with the correctly ordered items of that modality.
The gap between the system and the oracle interleave measures interleaving loss. The gap to the oracle within-order measures within-modality loss. We predict that interleaving loss dominates for raw CLIP and still accounts for a substantial share after GR-CLIP. If it does not, the remaining stages lose their motivation and we report that.
The second stage tests the scale hypothesis directly. Per query, we compare the variance of cosine scores for text candidates against image candidates, before and after mean subtraction. We then check whether that variance mismatch predicts the query's residual interleaving loss under GR-CLIP.
The third stage trains a hierarchy of calibrators that grow in capacity, all on the same frozen CLIP embeddings:
- A single global offset per modality.
- A global scale and offset per modality.
- The query-conditioned scale and offset described above, trained with the Plackett–Luce loss.
- A groupwise scoring network over mixed groups of size two and three, fed the centered embeddings together with modality-type indicators.
Training lists are built from MSCOCO and Flickr30k. Each query caption is paired with a pool of held-out captions, images, and image-caption pairs. Graded relevance comes from the dataset's paired annotations, and partial credit for near-misses is kept separate as an ablation, because the sentence-encoder proxy used by the smooth-NDCG paper would quietly turn the problem back into text-only matching. Models are then evaluated zero-shot on both MixBench domains. A small MixBench calibration split, of the same size GR-CLIP uses, serves as an in-domain variant.
Baselines
The naive reference point is raw CLIP cosine scoring over the mixed corpus with no correction. It gives the size of the problem.
The strongest existing method is GR-CLIP from Closing the Modality Gap for Mixed Modality Search. It is the only published method designed for this exact task, and it reports improving NDCG@10 by up to 26 percentage points over standard CLIP on MixBench. It also reports beating the generative embedding model VLM2Vec by about 4 percentage points while using 75 times less compute. The file gives no absolute NDCG@10 value for GR-CLIP, so we will rerun it ourselves. Our run will use the same CLIP backbone, the same calibration-set size, and its published fusion weights, re-tuned on the same validation queries we use for our model. Beating it means a statistically significant NDCG@10 improvement, measured with paired bootstrap over queries, on both the OVEN and VisualNews portions, with at most as much calibration data.
I0T_post from I0T: Embedding Standardization Method Towards Zero Modality Gap is the variance-aware post-hoc alternative. On Flickr30k it reports text-to-image accuracy rising from 67.10% to 76.30%, but it has no reported MixBench result on Lacuna. We run it under the same protocol. If standardization captures the scale term for free, it is the baseline most likely to tie our method.
ModGap from Closing the Modality Gap Aligns Group-Wise Semantics represents fixing the space instead of the ranker. It is retrained from the same CLIP initialization on the same MSCOCO data. No reported MixBench result was found on Lacuna for it.
The key-ingredient ablations are three:
- The query-conditioned head with conditioning removed, which is rung 2 of the hierarchy.
- The same head trained with a pointwise relevance loss instead of the listwise loss.
- The same head trained on single-modality lists only.
The third should fail if the claim is right, because the per-pool offset cancels and gives that objective no training signal.
Why not just…?
Why not just subtract modality means, as GR-CLIP does? It is the default, and a strong one. But it corrects only location, using a correction fixed after calibration. If the stage-two analysis finds no residual mismatch in score spread after centering, and rung 3 does not beat GR-CLIP, then mean subtraction is sufficient. The contribution would then shrink to the decomposition diagnostic, and we would say so.
Why not just fit one global temperature and offset per modality on the scores? This is the real threat. It is a few parameters and it targets interleaving directly. It is rung 2 of our hierarchy, so the experiment settles it. Our method only earns its place if query conditioning gives a significant gain over rung 2. The gain should be concentrated in queries whose spread mismatch is largest. A uniform small gain would not count.
Why not just fine-tune the encoders with a listwise loss, or close the gap with ModGap? Retraining costs compute and changes the embedding geometry. That geometry is the thing Liang et al. show can carry zero-shot and fairness benefits. We compare against both retraining routes directly. We also check zero-shot CIFAR-10 accuracy and FairFace misclassification rates for each, so that any ranking gain from retraining is weighed against what it costs elsewhere.
Why not just use a large vision-language embedding model? GR-CLIP already reports outperforming VLM2Vec on MixBench at a fraction of the compute. Scale alone therefore does not remove intra-modal bias. We include VLM2Vec only as a reference point.
Decisive Checks
The idea is supported if three things hold:
- Interleaving loss accounts for most of the gap between raw CLIP and the best achievable ordering.
- The query-conditioned listwise calibrator beats both GR-CLIP and the global affine calibrator on both MixBench domains.
- The groupwise network at rung 4 adds little on top of calibration.
Together these would show that mixed-modality ranking needs calibration rather than candidate-interaction modeling. The idea is refuted if the oracle-interleave analysis shows that within-modality loss dominates, or if rung 2 matches rung 3. Either outcome is informative for practitioners choosing among the existing fixes.
References
- Fast and Stable Maximum Likelihood Estimation for Incomplete Multinomial Models
- Closing the Modality Gap Aligns Group-Wise Semantics
- Closing the Modality Gap for Mixed Modality Search
- I0T: Embedding Standardization Method Towards Zero Modality Gap
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Integrating Listwise Ranking into Pairwise-based Image-Text Retrieval
- Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks