When a vision-language model such as CLIP or BLIP receives corrupted or out-of-domain queries, for example blurred photos or typo-ridden captions, their embeddings drift away from where clean queries of the same modality normally sit. Retrieval accuracy then collapses. The usual repair is test-time adaptation, meaning a few gradient steps on normalization parameters for every incoming batch. We propose a gradient-free alternative. We treat the corrupted embedding as a noisy copy of a clean one and apply the classical linear minimum-mean-square-error estimator (the "Wiener filter"), which pulls each query back toward the clean distribution using second-order statistics. The difficulty is that the filter needs the inverse of the test-batch covariance, and a batch of 64 queries in a 256- or 512-dimensional space gives a covariance estimate that is rank-deficient and very noisy.
The fix comes from wireless communications. In massive-antenna base stations, truncated polynomial expansion replaces a matrix inverse with a degree-3 to degree-6 polynomial in the matrix. Its few coefficients are chosen offline using random matrix theory, the mathematics of how eigenvalues of large random matrices behave. We reuse that machinery to choose the polynomial coefficients that minimize the expected reconstruction error given the batch size, rather than the error on the noisy sample covariance. The result is a test-time correction computed in closed form, with no learning rate and no backpropagation. It restores the source geometry of the query modality instead of erasing the gap between image and text embeddings.
Core Claim
Several papers link query shift to the modality gap, the systematic offset between image and text embeddings. Test-time Adaptation for Cross-modal Retrieval with Query Shift shows that corrupted queries widen this gap and lose intra-modal uniformity. It repairs both by iteratively tuning LayerNorm. Two other papers argue that the gap itself is partly useful. Mind the Gap shows the gap is created by contrastive training and that it matters for downstream behaviour. Is the Modality Gap a Bug or a Feature? shows the gap is an orthogonal displacement that buys noise robustness, but its analysis is limited to identity-covariance noise and leaves "general covariance structures" open. These findings motivate a different target: undo the shift while keeping the gap.
Assume a working model in which the corrupted query embedding $x = z + n$ is the clean embedding $z$ plus corruption $n$, uncorrelated with $z$. The optimal linear correction is then
$$\hat z = \mu_z + \Sigma_z\, \Sigma_x^{-1}(x - \mu_x).$$
Here $\mu_z, \Sigma_z$ are the mean and covariance of clean query embeddings, estimated offline from abundant source data. $\mu_x, \Sigma_x$ are the mean and covariance of the shifted test distribution. We replace $\Sigma_x^{-1}$ by $P(S)=\sum_{k=0}^{L} c_k S^k$, where $S$ is the sample covariance of the current test batch of $n$ queries and $L\le 4$.
The expected error $\mathbb{E}\|\Sigma_z P(S)(x-\mu_x) - (z-\mu_z)\|^2$ is a quadratic function of the coefficient vector $c$. Its entries are normalized traces such as $\mathrm{tr}(\Sigma_z S^j \Sigma_x S^k \Sigma_z)/d$, where $d$ is the embedding dimension. Random matrix theory gives deterministic approximations to these traces in terms of the moments of $\Sigma_x$, and those moments can be recovered from the observed traces $\mathrm{tr}(S^k)/d$. Solving for $c$ therefore costs one $(L{+}1)\times(L{+}1)$ linear solve per batch.
The claim is specific. When the batch size is comparable to or smaller than the dimension, coefficients chosen this way should beat two alternatives at the same second-order target. The first is the exact filter computed with standard covariance shrinkage. The second is a ridge-regularized inverse with a tuned regularizer. This mirrors the finding of Linear Precoding Based on Polynomial Expansion: Large-Scale Multi-Cell MIMO Systems that coefficient-optimized polynomials outperform regularized zero-forcing when channel estimates are imperfect. A noisy sample covariance is our analogue of imperfect channel knowledge.
We do not claim a speed advantage as the main contribution. At $d\le 768$, an exact eigendecomposition takes milliseconds, so the cubic-cost argument that motivates polynomial expansion in base stations is weak here. The polynomial form matters because it turns the finite-sample-aware choice of a spectral filter into a closed-form quadratic problem. The streaming matrix-vector evaluation, which costs $O(L d n)$, is a side benefit.
Research Program
The method borrows from three sources, each in a distinct role.
The estimator structure comes from linear detection in massive MIMO. Linear precoding based on polynomial expansion: reducing complexity in massive MIMO shows that degree 3 nearly matches regularized zero-forcing. Asymptotic Moments for Interference Mitigation in Correlated Fading Channels shows that degree 6 nearly matches the full linear minimum-mean-square-error detector under correlated fading. It also supplies recursive formulas for asymptotic moments, which we adapt to the case where a known matrix $\Sigma_z$ is sandwiched between powers of $S$.
The same paper constrains the design. Higher moments converge slowly at moderate dimensions, so we cap $L$ and expect the optimal degree to fall as the batch shrinks. We report this degree-versus-batch-size curve as a result in its own right.
The problem statement comes from the test-time adaptation paper above, which also supplies the benchmarks.
The build proceeds in four steps. First, estimate $\mu_z$ and $\Sigma_z$ for each frozen encoder from clean source queries: COCO and Flickr30k training images for image queries, and captions for text queries. Second, for each test batch, compute the trace moments of $S$, map them to population moments via the inverse Marchenko-Pastur moment relations, form the quadratic objective, and solve for $c$. Third, apply the correction to each query. Because $x$ is in the batch, we use a leave-one-out version of $S$ to avoid self-bias. Fourth, retrieve against the unchanged gallery.
Two diagnostic analyses accompany the main results. The first tests the additive model: we fit it on paired clean and corrupted embeddings from validation data and measure how much of the corrupted covariance it explains. Corruptions with poor fit tell us where a linear filter should fail. The second tracks the centroid distance between modalities and the uniformity within each modality before and after correction. This checks that we restore the source gap rather than collapse it.
Baselines
The naive reference is the frozen model with no correction. On COCO-C image shift with BLIP, the test-time adaptation paper reports Recall@1 of 45.0%.
The first-order baseline is batch mean re-centering in the spirit of I0T_post from I0T. That method subtracts modality means and re-normalizes, cutting the Flickr30k centroid distance from 0.7642 to 0.0102 and raising I2T from 69.60% to 73.30% and T2I from 67.10% to 76.30%. We adapt it to re-center test queries to the clean source mean, which equals our method with $P=I$ and $\Sigma_z$ dropped. It isolates how much the second-order term adds.
The key ablation is the exact Wiener filter computed with a shrinkage covariance estimate (Ledoit-Wolf) via eigendecomposition. A second variant uses a ridge inverse $(S+\rho I)^{-1}$ whose $\rho$ is tuned by an oracle on held-out corrupted data. Both target the same second-order quantity with a different spectral estimator. Our method must beat them, not merely tie, at small batch sizes. No reported result for either on these benchmarks was found on Lacuna, so we run them ourselves.
A second ablation fits the polynomial coefficients to the raw sample covariance without the random-matrix correction. It isolates whether the finite-sample correction is the active ingredient.
The strongest existing method is TCR from Test-time Adaptation for Cross-modal Retrieval with Query Shift. It reports BLIP Recall@1 of 59.1% on COCO-C image shift. We run its released configuration on the same BLIP backbone, corruption splits, and batch sizes. We report both its accuracy and its wall-clock adaptation cost. Beating it means matching its Recall@1 within one point at a fraction of its runtime, or exceeding it outright. We expect to win clearly only on corruptions where the additive model fits well.
Why not just…?
Why not just re-center the test batch to the clean mean? Mean shifts are where the gap visibly lives. However, AlignCLIP reports that naive shifting distorts intra-modal structure. I0T shows that the modality signature is carried by specific peak coordinates rather than only by the centroid. Corruptions such as blur or noise also inflate variance along particular directions, which a translation cannot undo. The first-order baseline measures this directly. If it captures nearly all of the gain, the second-order machinery is not justified.
Why not just compute the exact filter with a shrunk covariance? This is the real threat. At these dimensions, an eigendecomposition is cheap, and any spectral filter a polynomial can express, an eigendecomposition can express too. Our only argument is that the coefficients are chosen to minimize the expected reconstruction error at the actual batch-to-dimension ratio. Ledoit-Wolf, by contrast, minimizes covariance-estimation error without reference to the downstream filter. A tuned ridge has one degree of freedom. The batch-size sweep settles the question. If the shrinkage or ridge filter matches us at every ratio, we will report that the contribution reduces to the Wiener-filter formulation of gradient-free test-time adaptation, with the polynomial as an implementation choice.
Why not just tune LayerNorm at test time, as TCR does? TCR is the accuracy bar, and it can model nonlinear shifts that a linear filter cannot. Tuning LayerNorm in Attention shows such tuning works but is highly sensitive to learning rate and number of epochs. TCR also needs backpropagation through the encoder for every batch. Our method has no step size, so it cannot diverge. The per-corruption comparison will show where linear correction suffices and where it does not.
Why not just retrain with a gap-closing objective? Closing the Modality Gap Aligns Group-Wise Semantics raises MSCOCO V-Measure from about 13.0 to 23.63. This requires retraining, does not address test-time corruption, and deliberately removes the gap that Is the Modality Gap a Bug or a Feature? links to robustness. Retraining answers a different question, so we do not treat it as a competitor for query shift.
Proposed Experiments
The main experiment uses COCO-C and Flickr-C, which cover 16 image corruptions and 15 text corruptions. We test BLIP, to compare with TCR, and CLIP ViT-B/32 and ViT-L/14. Batch sizes are 16, 32, 64, 128, 256, and 512, so the batch-to-dimension ratio spans roughly 0.03 to 2. We report Recall@1, @5, and @10 per corruption and averaged. No reported result was found on Lacuna for Flickr-C or for text-side shift, so all those numbers will be new.
The first decisive check is the batch-size curve. Our method should beat the shrinkage-Wiener and oracle-ridge baselines when the batch is smaller than the dimension, and converge to them when the batch is much larger. If it fails to do so, we should drop the core claim.
The second decisive check is gap preservation. After correction, the image-text centroid distance should return toward its clean-data value, not toward zero. Clean-data retrieval on Flickr30k and MSCOCO should be unchanged, because the filter should be close to the identity when there is no shift.
The third check is robustness on ImageNet under Gaussian noise. There, Is the Modality Gap a Bug or a Feature? reports gains of up to 10% from its translation method with negligible clean-accuracy loss. We compare our filter against that translation at matched noise levels.
Finally, we report the fitted coefficients, the optimal degree $L$ at each batch size, and per-batch latency against TCR. The aim is for readers to see when a three-coefficient linear correction is enough and when a corruption requires gradient-based adaptation.
References
- Test-time Adaptation for Cross-modal Retrieval with Query Shift
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Is the Modality Gap a Bug or a Feature? A Robustness Perspective
- Linear Precoding Based on Polynomial Expansion: Large-Scale Multi-Cell MIMO Systems
- Linear precoding based on polynomial expansion: reducing complexity in massive MIMO
- Asymptotic moments for interference mitigation in correlated fading channels
- I0T: Embedding Standardization Method Towards Zero Modality Gap
- Mitigate the Gap: Improving Cross-Modal Alignment in CLIP
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
- Closing the Modality Gap Aligns Group-Wise Semantics