Design of the topology for contrastive visual-textual alignment

Zhun Sun

TL;DR

Standard contrastive vision-language models like CLIP map representations onto a unit hypersphere and rely on a learnable softmax temperature to counteract noise in web-scale datasets. This paper shows that replacing the hypersphere with an oblique manifold (a Riemannian product of multiple unit spheres) broadens the distance range and relaxes metric constraints, requiring only a two-line code modification while outperforming officially released CLIP models.

What Problem Does This Paper Address?

Web-scale pre-training datasets for vision-language models (e.g., CLIP) are notoriously noisy. Semantically related image-text pairs are frequently marked as negative pairs (false negatives), and loosely related pairs are marked as positive (false positives)—a challenge termed semantic ambiguity.

Under this noise, models do not achieve perfect alignment; instead, they converge to an equilibrium where attractive and repulsive contrastive gradients balance out. Because contrastive loss uses softmax cross-entropy, reaching this numerical equilibrium requires logits that exceed the natural $[-1, 1]$ range of cosine similarity on a unit hypersphere. Existing models patch this issue with a learnable temperature parameter $\tau$ to scale the distance range. However, the authors find that the model prioritizes optimizing $\tau$ before learning rich semantic features, delaying convergence and introducing numerical instability during low-precision training.

Methods

Rather than relying heavily on artificial temperature scaling, the authors redesign the embedding topology:

  • The Oblique Manifold $\text{Ob}(n, m)$: The embedding space is structured as an oblique manifold, which represents $m \times n$ matrices with unit-norm columns. Geometrically, this corresponds to the Riemannian product manifold of $m$ spheres: $\mathcal{S}^{n-1} \times \dots \times \mathcal{S}^{n-1}$.
  • Negative Inner Product Distance: Instead of geodesic distances (which enforce strict triangle inequalities), the similarity is measured as the negative trace of the matrix product: $$d(u, v) = -\text{tr}(u^\top v)$$ This expands the natural distance range from $[-1, 1]$ to $[-m, m]$ while preserving $O(b^2)$ memory footprint during mini-batch matrix multiplication.

  • Two-Line Implementation: In standard CLIP, the $d$-dimensional output vector is simply reshaped into $m$ sub-vectors of dimension $n$, $\ell_2$-normalized along the sub-vector dimension, and flattened back.

  • Multi-Token Implementation: To give each sub-sphere unique expressive capacity, the authors introduce multiple randomly initialized [CLS] tokens into the Transformer backbone, treating each token output as an independent sub-sphere on the oblique manifold.

Datasets

  • Pre-training:
  • The YFCC-15M subset (~15.3 million pairs) for controlled ablation studies.
  • A scaled web-crawled dataset combined with public sources (Conceptual Captions, COCO, Visual Genome, Open Images, RedCaps, etc.), totaling ~420M images and ~500M image-text pairs.
  • A 4M multi-dataset benchmark (CC3M, COCO, VG, SBU) for testing integration with the Triple Contrastive Learning (TCL) framework.
  • Evaluation:
  • Flickr30k and MS-COCO for zero-shot text-to-image and image-to-text retrieval.
  • ImageNet-1K for zero-shot and linear-probe classification.
  • RedCaps as an out-of-domain dataset to evaluate feature distance distributions.

Key Insights and Ideas

  • The Temperature Trap: Softmax temperature is essentially a proxy for dataset noise level. When temperature is learnable, the network aggressively converges $\tau$ early in training, stalling visual-textual representation learning.
  • The Need for "Relaxed" Triangle Inequality: Strict metrics (like Euclidean $\ell_2$ or manifold geodesic distance) obey $d(u_2^*, v_1^*) \le 3\epsilon^+$. When false negatives exist, this bounds negative pairs too tightly to positive pairs. The negative inner product breaks strict metric properties, offering a relaxed geometric penalty that tolerates semantic noise.
  • Preserving Uniformity: Unlike unbounded Euclidean space (where features often explode and lack a tractable uniform prior), the oblique manifold retains a well-defined uniform distribution via surface area measure.

Key Results

  • YFCC-15M Benchmark (ViT-S/16):
  • $\text{Ob}(64, 8)$ with negative inner product achieved 52.3% Image-to-Text R@1 and 32.89% Text-to-Image R@1 on Flickr30k—a +4.0% and +1.44% improvement over the spherical baseline.
  • When temperature was fixed ($\tau=1.0$), the spherical baseline collapsed (5.1% I2T R@1), while the oblique manifold retained 30.3% I2T R@1, demonstrating robustness to scale without temperature fitting.
  • Large-Scale Benchmark (~500M pairs):
  • Equipping a ViT-B/16 model with the multi-token configuration $\text{Multi}(16, 32)$ achieved 76.4% zero-shot top-1 accuracy on ImageNet-1K, outperforming OpenAI's officially released ViT-B/16 (68.7%) and even their larger ViT-L/14 (75.5%).
  • On Flickr30k zero-shot retrieval, it attained 85.2% I2T R@1 and 66.3% T2I R@1 with less than 8% additional computational overhead.

Limitations and Future Work

  • Compute Limitations: Constraints prevented testing scaling laws on extremely large batch sizes (>32k) and giant parameter regimes (e.g., ViT-H or ViT-G).
  • Diminishing Returns on Multi-Loss Architectures: When evaluated within frameworks like TCL—which incorporate masked language modeling, image-text matching heads, and cleaner datasets—the gains from changing the contrastive alignment topology were marginal.
  • Token Embeddings: The work concentrates solely on global pooled/[CLS] embeddings; designing non-spherical topologies for dense, dense-patch, or full token sequences remains open for future research.

Summary updated

Abstract

Pre-training weakly related image-text pairs in the contrastive style shows great power in learning semantic aligning cross-modal models. The common choice to measure the distance between the feature representations of the image-text pairs is the cosine similarity, which can be considered as the negative inner product distance of features embedded on a sphere, mathematically. However, empirically, aligning image-text pairs on the spherical topology is vulnerable to the semantic ambiguity phenomenon resulting from the noise in the pre-training datasets. Specifically, under the noisy training data, instead of the optimal alignment-uniformity solution, the system would achieve an equilibrium (a gap between distances of positive and negative pairs), when the gradients for attraction and repulsion are neutralized. Although intuitively, the model should always find this equilibrium given a sufficiently long training scheme, its numerical values might be out of the distance range (e.g. [-1, 1] for the cosine similarity). In the practice of former studies, this problem is partly tackled by introducing a learnable softmax temperature parameter, in other words, by explicitly scaling the range of the distance function. In this work, we alternatively design the topology of embedding space and its endowed distance function. Motivated by studies that make use of Riemannian geometry for visual tasks, we propose a rather simple solution to address the aforementioned equilibrium problem. That is, we map the feature representations onto the oblique manifold endowed with the negative inner product as the distance function. In the experimental analysis, we show that we can improve the baseline performance by a large margin (e.g. 4\% in the zero-shot image to text retrieval task) by changing only two lines of the training codes.
BibTeX
@inproceedings{sun2023design,
  author = {Zhun Sun},
  title = {{Design of the topology for contrastive visual-textual alignment}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=UyC1dXUA-n},
}
Figure