The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model

Zijie Yu, Gaowen Liu, Ramana Rao Kompella, Philip S. Yu, Yue SongarXiv 2026 · arXiv

TL;DR

Contrastive Language-Image Pretraining (CLIP) models map data to a hyperspherical latent space, yet existing probabilistic models treat this space as a single, flat Gaussian distribution. This paper introduces MovMF-CLIP, a framework that models the latent space as a Mixture of von Mises-Fisher distributions on a unit hypersphere. This geometrically consistent approach prevents rare semantic concepts from being misclassified as out-of-distribution, while enabling fast, interpretable semantic decomposition.

What Problem Does This Paper Address?

CLIP aligns images and text using cosine similarity, which naturally creates a hyperspherical embedding space. However, real-world CLIP embeddings suffer from structural irregularities like the "modality gap" and the "cone effect" (where embeddings cluster into narrow, anisotropic regions).

To perform probabilistic reasoning or detect anomalies in this space, previous methods apply a whitening transform and assume the data follows a single global Gaussian distribution. The authors point out a fatal flaw in this unimodal assumption: it conflates semantic rarity with distributional abnormality. Because rare, long-tail concepts lie far from the global mean, a single Gaussian assigns them a very low likelihood, mistakenly treating valid but rare data exactly like true out-of-distribution (OOD) anomalies.

Figure 1: Density modeling of CLIP latent space (visualized via dimensionality reduction on real data). (a) Gaussian-based approaches such as W-CLIP model the latent space with a single global distribution, which can assign low likelihood to valid long-tail concepts due to their distance from the global mean. (b) MovMF-CLIP models the space as a hyperspherical semantic mixture, capturing the multimodal structures and providing density estimates aligned with semantic clusters.

Methods

The authors propose MovMF-CLIP, a three-stage pipeline designed to respect both the directional and multimodal nature of CLIP:

  1. Geometric Calibration via Whitening: They apply a covariance-based whitening transformation to strip away nuisance anisotropy and dataset bias. Crucially, they do not assume the resulting space is Gaussian; whitening is used strictly as a metric normalization step.
  2. Hyperspherical Normalization: The whitened features are projected onto a unit hypersphere ($\mathbb{S}^{d-1}$), aligning the probabilistic model with CLIP's native cosine-similarity geometry.
  3. Mixture of von Mises-Fisher (MovMF) Fitting: Instead of a single distribution, they model the latent space using a mixture of independent vMF distributions (the spherical analog of Gaussians). Each mixture component acts as a coherent "semantic prototype." They use the Expectation-Maximization (EM) algorithm to efficiently learn the parameters, employing a fast numerical approximation to handle high-dimensional concentration parameters ($\kappa$).
Figure 2: Overview of the MovMF-CLIP Framework. We first extract raw embeddings using the CLIP encoder, which exhibit severe anisotropy. To address this, we apply geometric calibration via whitening (˜z = W(z −µ)) and normalize the features onto a unit hypersphere (u = ˜z/∥˜z∥2). Finally, we fit a MovMF distributions on the hypersphere using the EM algorithm, yielding a prin- cipled multimodal density model.

Datasets

  • MS-COCO 2017: Used as the primary in-distribution (ID) dataset for density estimation, validation, and long-tailed fairness analysis.
  • OpenImages: A cleaned subset (removing overlapping categories with MS-COCO) used to evaluate out-of-distribution (OOD) detection performance.
  • LAION-400M: The pre-training dataset for the OpenCLIP ViT-B/32 backbone used in the semantic decomposition experiments.

Key Insights and Ideas

  • Whitening $\neq$ Gaussianity: Whitening is often used to justify a Gaussian likelihood. Here, the authors decouple the two, using whitening purely to fix the metric space while allowing the underlying distribution to remain explicitly multimodal.
  • Decoupling Rarity from Anomaly: By giving rare concepts their own dedicated mixture components, MovMF-CLIP assigns them calibrated, realistic likelihoods rather than penalizing them for being far from a global average.
  • Intrinsic Interpretability: Because each mixture component represents a semantic concept, an embedding's posterior probabilities act as a natural, sparse semantic decomposition. You don't need to train auxiliary decoders or rely on external concept dictionaries; the model explains its own representations.

Key Results

  • Superior OOD Detection: On MS-COCO vs OpenImages, MovMF-CLIP reduced the False Positive Rate at 95% TPR (FPR95) from 67.76% (W-CLIP) to 48.00%. On long-tail subsets, the improvement was even more dramatic, slashing FPR95 from 75.05% to 33.48%.
  • Fairer Likelihoods: In separating head vs tail concepts, MovMF-CLIP achieved an AUROC of 0.5819 (where 0.5 is perfectly fair/unbiased), vastly outperforming W-CLIP's biased score of 0.7826.
  • Fast Semantic Decomposition: MovMF-CLIP achieved the highest Semantic Relevance score (0.673) against competitive baselines like SPLICE, while delivering a $13\times$ speedup in inference time (9.8 ms vs 132.7 ms per image) because it relies on closed-form posterior calculations rather than iterative optimization.
  • Mitigating Generative Drift: When inserted into an iterative CLIP-UnCLIP image generation loop, MovMF-CLIP acted as a geometric anchor, preventing the semantic degradation and mode collapse common in standard pipelines.

Limitations and Future Work

  • Flat Mixture Structure: The current model assumes a flat mixture with a fixed number of components ($K$). However, human language and visual concepts are inherently hierarchical (e.g., a "husky" is a "dog", which is an "animal").
  • Future Directions: The authors suggest extending the framework to hierarchical mixture models (like Hierarchical Dirichlet Processes) to allow the number of components to adapt dynamically. They also highlight potential connections to Riemannian score-based generative models to provide geometry-aware guidance in diffusion pipelines.

Summary updated

Abstract

Contrastive Language-Image Pretraining (CLIP) representations form a semantic embedding space governed by cosine similarity, reflecting an intrinsic hyperspherical geometry. However, existing probabilistic interpretations typically rely on Gaussian assumptions, which fail to capture this directional and multimodal structure. We propose a principled density model for the CLIP latent space based on Mixtures of von Mises-Fisher (MovMF) distributions defined on the unit hypersphere. Using the Expectation-Maximization (EM) algorithm, we efficiently learn a probabilistic model in which each mixture component corresponds to a coherent semantic concept. This formulation yields a closed-form likelihood naturally aligned with hyperspherical geometry, enabling accurate and interpretable density estimation. Empirically, our model significantly improves long-tailed and out-of-distribution detection and provides a natural semantic decomposition, representing each embedding as a sparse probabilistic combination of interpretable concepts. These results suggest that CLIP latent space is more faithfully characterized as a hyperspherical semantic mixture rather than an isotropic Gaussian, establishing a simple and geometrically consistent probabilistic framework for modeling and understanding multimodal representations. Project page is available at https://xiaoyuzhizi.github.io/movmf-clip/.
BibTeX
@misc{yu2026hyperspherical,
  author = {Zijie Yu and Gaowen Liu and Ramana Rao Kompella and Philip S. Yu and Yue Song},
  title = {{The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model}},
  year = {2026},
  eprint = {2607.13660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.13660},
}
Figure