Vision-language models such as CLIP embed images and captions into one space, yet the two kinds of embeddings sit in separate clusters. This separation is called the modality gap. The field disagrees about what the gap is. One camp says it is a slow optimization transient that would disappear with enough training. A second says contrastive training produces it as a stable end state. A third says it is a useful buffer that should be kept. These positions call for different fixes, and the literature currently settles the question only by training longer or by intervening and watching what happens. We propose to answer it from a short training run.
The tool is the Kurdyka-Łojasiewicz (KL) inequality, a classical result from non-convex optimization. It links how far a loss is above its eventual limit to how steep the loss still is. When the inequality holds, the total distance an optimizer will still travel is bounded by a quantity we can measure. That bound tells us whether the remaining trajectory is long enough to close the gap. We apply this to contrastive training and turn it into a forecast of the limiting gap. We validate the forecast against long ground-truth runs and use it to predict which gap-reduction method will work in which training regime.
Core Claim
The claim has three parts. The first is a structural diagnosis. Yaras et al. prove that under gradient flow with a learnable temperature the gap shrinks only at rate $\Omega(1/\log(t)^2)$ (Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning). The temperature is the scale that sharpens the softmax over similarities. A decay this slow is what one expects when the objective's infimum lies "at infinity," here because the inverse temperature keeps growing. In that situation the iterates are unbounded and the finite-path-length argument of KL theory does not apply.
The second part follows from standard results. If the temperature is fixed or bounded, the contrastive loss on normalized embeddings is a real-analytic function on a compact set. Analytic functions satisfy the Łojasiewicz inequality, so bounded gradient-descent iterates have finite length and converge to a single critical point. This is the classical theorem of Attouch et al. (Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality). In that regime the gap does not creep toward zero indefinitely. It converges to a definite limit, which may or may not be zero, at a rate set by the local exponent. We have not found either point stated for contrastive objectives in the file.
The third part is the new instrument: a KL-based estimator of the local exponent and of the limiting gap, computed from checkpoints of a finite run. The KL inequality for a loss $f$ near a limit value $f_\infty$ reads
$$\phi'\big(f(x)-f_\infty\big)\,\|\nabla f(x)\| \ge 1, \qquad \phi(s)=c\,s^{1-\theta},$$
where $x$ are the parameters, $c>0$ is a constant, and $\theta\in[0,1)$ is the Łojasiewicz exponent. Smaller $\theta$ means the loss stays steep near its limit. Given this inequality, the classical descent argument bounds the remaining path of the iterates $x_k$:
$$\sum_{j\ge k}\|x_{j+1}-x_j\| \;\le\; C\,\big(f(x_k)-f_\infty\big)^{1-\theta} + \|x_k-x_{k-1}\|,$$
where $C$ depends on step size and smoothness. The gap is a Lipschitz function of the embeddings of a fixed probe set, so the same quantity bounds how much the gap can still change. We estimate $\theta$ and $f_\infty$ by fitting the log-log relation between suboptimality and gradient norm across checkpoints. That produces a forecast interval for the final gap and a regime label: linear convergence for $\theta\le 1/2$, polynomial convergence for $\theta>1/2$, and effectively no bounded limit when the fitted $\theta$ drifts toward 1.
Research Program
The theory lives in a minimal model: free image and text embeddings on the unit sphere, trained with InfoNCE (the standard contrastive loss, which pulls matched pairs together and pushes mismatched pairs apart). This is the analogue of the gradient-flow setting of Yaras et al. In this model we will prove three things. With fixed temperature, gradient descent converges to one critical point. Zero-gap configurations are critical points only under stated uniformity conditions on the data. We will also compute or bound $\theta$ at the zero-gap critical points and at the gapped ones. That last question connects directly to the result of Fahim et al. (Its Not a Modality Gap: Characterizing and Addressing the Contrastive Gap). They show that even identical encoders trained on identical data end up gapped because of low uniformity on the hypersphere. The model tells us whether that gapped state is a stable limit with a well-behaved exponent, which is how their stationary "contrastive gap" story would look, or a slow transient, which is how Yaras et al.'s story would look.
For the learnable-temperature case we will make precise the claim that unbounded inverse temperature breaks boundedness. We then show that capping the logit scale restores a KL rate. Real networks train with stochastic minibatches, so their iterates are noisy. To cope with that, we evaluate loss and gradient on a fixed large probe batch at each checkpoint. We also borrow the $\epsilon$-approximate KL notion of Gu et al. (Inexact Proximal Gradient Methods for Non-Convex and Non-Smooth Optimization). Their notion tolerates summable per-step errors and supplies the error budget our forecast interval must absorb.
Empirically, we train small two-encoder models from scratch on Microsoft COCO, mirroring Yaras et al.'s setup, under four regimes: learnable temperature, capped temperature, fixed temperature, and temperature scheduling. We also fine-tune pretrained CLIP on COCO and Flickr30k. Each regime is run far past its usual budget, several times the standard number of epochs, so that we have a ground-truth late-training gap. We measure the gap as centroid distance, following I0T: Embedding Standardization Method Towards Zero Modality Gap and Fahim et al. The forecaster sees only checkpoints from the first 10–30% of each run.
The practical output is a prescription. If the forecast says the run converges to a gapped critical point, changing the temperature cannot help at that fixed point, and a regularizer that changes the critical set is needed, such as the centroid-uniformity term of ModGap (Closing the Modality Gap Aligns Group-Wise Semantics). If the forecast says the run is in the unbounded, slow-drift regime, temperature control should suffice. Liang et al. (Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning) show the gap's size affects fairness and zero-shot accuracy. The forecast therefore also tells a practitioner which gap they will end up with before paying for the full run.
Baselines
The naive reference point assumes the gap measured at the forecasting checkpoint is the final gap. A good forecaster must beat this on runs where the gap is still moving, while matching it on runs that have already converged.
The strongest forecasting competitor is direct curve extrapolation of the gap time series itself. We fit three families and let each run pick its best by held-out checkpoint error: exponential decay to an offset, power-law decay to an offset, and the $a + b/\log(t)^2$ form implied by Yaras et al. It uses exactly the same checkpoints as our estimator and gets the same fitting budget. For limiting-gap forecasting, no reported result was found on Lacuna, so this baseline is built here. Beating it means lower absolute error in predicted final centroid distance, averaged across regimes and seeds. It also means better-calibrated intervals, and correctly classifying runs as converging-to-zero versus converging-to-gapped, where the naive fit misclassifies.
The ablation that isolates the key ingredient is our estimator stripped of gradient-norm information, so it fits only the loss curve. If this does as well as the full estimator, the KL coupling between steepness and suboptimality adds nothing. A second ablation drops the $\epsilon$-approximate noise correction to test whether it matters under minibatch noise.
For the prescriptive claim, the competitors are the published interventions, each run in every regime with matched compute and tuning budgets. Temperature scheduling from Yaras et al. reduced the gap from 0.294 to 0.088 and raised MS COCO image-to-text retrieval from 78.4% to 85.3%. ModGap reduced the gap from 0.47 to 0.03 and reached MS COCO clustering V-Measure 23.63 (up from about 13.0), with MSR-VTT V-Measure 32.1 versus 23.3 for CLIP. Post-hoc centering, I0T_post, reduced centroid distance from 0.7642 to 0.0102 and reached Flickr30k image-to-text retrieval of 73.30%. Our forecast "beats" blanket application of any one of these if it predicts in advance which one reaches the lower final gap and better downstream score in each regime, more often than always picking the single best-on-average method.
Why not just…?
Why not just train longer and look? Long training is our ground truth, so we are not claiming it is wrong. We are claiming it is expensive, and at CLIP scale it is unaffordable. The contribution is predicting its outcome from a fraction of the compute. If our forecasts are no better than the naive frozen-gap assumption, the method has failed, and the experiments will show this directly.
Why not just fit a curve to the gap? This is the real threat. Yaras et al. already supply a functional form, and for well-behaved runs a curve fit may be enough. Our estimator has one potential advantage. It uses the steepness of the loss, which shows early whether the trajectory is approaching a bounded critical point or drifting. The gap curve alone can look identical in both cases for a long time. The head-to-head comparison and the loss-only ablation settle this. If curve fitting matches us, we will report that the KL framing contributes only the regime theory (the boundedness diagnosis and the fixed-temperature convergence theorem), not a better forecast.
Why not just center the embeddings after training? I0T_post zeroes the centroid gap at no training cost. But it needs a batch at test time to estimate modality means. It also translates clusters rigidly rather than repairing the intra-modal distortion documented in Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion. Most importantly, it does not answer what training converges to, which is the question this proposal addresses. We include centering as a downstream competitor. If centered checkpoints match the regularized ones on clustering V-Measure and retrieval, the prescriptive half of the proposal loses its value, and we will say so.
Why not just use temperature scheduling everywhere? Our theory predicts a specific limit on this fix. In a regime with bounded temperature that converges to a gapped critical point, scheduling cannot remove a gap that the stationary point itself contains. The regime-by-intervention grid tests exactly this prediction.
Decisive Checks
The first check is in the free-embedding model, where the exponent can be computed. The estimated $\theta$ must recover the analytically derived value. The learnable-temperature runs must show the fitted exponent drifting toward 1, while capped-temperature runs show a stable $\theta$.
The second check is on COCO training runs. Forecasts made from the first 20% of checkpoints must predict the late-training centroid distance with lower error than the best curve fit, across at least five seeds per regime. They must also correctly separate runs that close the gap from those that settle at a nonzero gap.
The third check is the intervention grid. The forecast-guided choice between temperature scheduling and a ModGap-style regularizer must match or exceed the single best fixed choice. The measures are final gap, COCO retrieval R@1, clustering V-Measure, and CIFAR-10 zero-shot accuracy.
If the fixed-temperature runs show gaps that keep drifting with no stable exponent, our boundedness diagnosis is wrong, and the stationary contrastive-gap view loses support in these regimes. If they converge to gapped limits that our estimator anticipates, the dispute between the camps described above gets a mechanistic, testable answer.
References
- Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning
- Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Lojasiewicz Inequality
- Its Not a Modality Gap: Characterizing and Addressing the Contrastive Gap
- Inexact Proximal Gradient Methods for Non-Convex and Non-Smooth Optimization
- Microsoft COCO: Common Objects in Context
- I0T: Embedding Standardization Method Towards Zero Modality Gap
- Closing the Modality Gap Aligns Group-Wise Semantics
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion