Introduction
Contrastive vision-language models, such as CLIP, have become cornerstones of modern AI. These models typically operate by mapping image and text embeddings onto a unit hypersphere, where similarity is measured via cosine distance. However, this standard approach faces a significant hurdle: web-scale training data is inherently noisy. "False negatives"—pairs that are semantically related but labeled as non-matches—are common. On a unit sphere, the distance range is strictly constrained to $[-1, 1]$, forcing models to rely on a "learnable temperature" parameter ($\tau$) to scale these values and manage the resulting gradients. This paper argues that the unit sphere is a sub-optimal topology for noisy data and proposes a shift to the oblique manifold.
Methods: From Spheres to Oblique Manifolds
The core proposal is to replace the single unit sphere with an oblique manifold, which can be conceptualized as a product of multiple smaller spheres. In a Transformer-based architecture (like ViT or BERT), this is elegantly implemented by utilizing multiple [CLS] tokens instead of just one.
Each of the $m$ [CLS] tokens represents a point on one of the sub-spheres. The similarity between a query and an image is then calculated as the sum of the inner products across all corresponding token pairs. This simple architectural change expands the potential distance range from the narrow $[-1, 1]$ to a much broader $[-m, m]$.
Projecting onto this new manifold is computationally efficient, requiring only a column-wise $L_2$ normalization. This allows the model to scale its representational capacity without significant overhead.
Key Insights: Geometry and Noise
The research highlights that the learned temperature parameter in CLIP acts as a "noise gauge." In noisy environments, the model requires a wider numerical range to reach an equilibrium between the attractive forces of true positives and the repulsive forces of false negatives.
A critical theoretical advantage of the oblique manifold is its "relaxed" triangle inequality. Unlike standard metric spaces (like Euclidean space) that strictly enforce distance relationships, the oblique manifold's non-metric distance function allows the model to be less "punished" when encountering semantically similar samples that are mislabeled as negatives.
As shown in the comparison above, the oblique manifold maintains desirable properties like a uniform surface measure while providing the necessary "relaxed" inequality and expanded distance range to handle real-world data noise effectively.
Results and Performance
The transition to a multi-token oblique implementation yields substantial empirical gains. A model using 16 [CLS] tokens achieved a 6.1% average improvement in zero-shot classification across benchmarks including ImageNet-1K and its variants (V2, Rendition, and Adversarial).
The improvements were most pronounced on challenging datasets like ImageNet-A (Adversarial), suggesting that the expanded topology provides better robustness. Furthermore, the models demonstrated faster convergence, reaching optimal alignment more quickly than spherical baselines because the topology naturally accommodates the necessary distance distributions.
Conclusion
By rethinking the underlying geometry of embedding spaces, this work demonstrates that the "temperature trick" in CLIP is a symptom of topological limitations. Moving to an oblique manifold via multiple [CLS] tokens provides a more expressive and noise-tolerant framework for visual-textual alignment, paving the way for more robust and efficient contrastive learning models.
Summary updated