Provable Cross-Modal Transfer of Hierarchical Concept Structure in CLIP Embeddings

Nghia Nguyen, Tianjiao Ding, Lachlan Ewen MacDonald, René VidalarXiv 2026 · arXiv

TL;DR

This paper provides a theoretical and empirical explanation for why CLIP models implicitly learn semantic hierarchies (like "a dog is a mammal") despite having no explicit taxonomic objective. The authors show that hierarchical structure is natively present in image data and "transfers" to text embeddings during training because images are naturally captioned at multiple levels of generality (e.g., an image of a Labrador being labeled as "Labrador," "dog," and "animal").

What Problem Does This Paper Address?

Contrastive vision-language models like CLIP are designed to align images with their corresponding text descriptions. They aren't given a syllabus on biology or linguistics, yet their internal representations (embeddings) reflect sophisticated taxonomies. For example, the embedding for "cat" is geometrically closer to "mammal" than to "car."

While researchers have observed this phenomenon, we haven't had a principled explanation for its origin. Why does a non-hierarchical objective—simply matching pairs—result in a hierarchical geometric structure? Furthermore, there is a curious asymmetry: image encoders seem to possess some of this structure even before training (at random initialization), whereas text encoders only acquire it after seeing data. This paper seeks to formalize this "Hierarchical Angular Separation" and prove how it propagates from one modality to the other.

Methods

The authors introduce several key frameworks to study this problem:

  • Hierarchical Angular Separation (HAS): A new geometric metric. A set of embeddings satisfies HAS if every concept is more similar (has a higher cosine similarity) to its descendants than to any "off-branch" concepts. They quantify failures of this condition using a metric called HAS-Viol.
  • Unconstrained Features Model (UFM): To make the complex math of deep learning tractable, the authors use UFM, which focuses on optimizing the last-layer features directly rather than the entire neural network.
  • Theoretical Analysis of Losses: They analyze both a simplified Linear Contrastive Loss (which allows for a closed-form solution) and the standard InfoNCE Loss used in CLIP.
  • The Mechanism of Hierarchical Co-occurrence: They model the fact that in real-world datasets, a specific image (a leaf in a taxonomy) is often paired with captions of varying specificity (ancestors in the taxonomy).
Figure 1: How contrastive learning transfers hierarchy from image to text embeddings. Top-left: the image prototypes are arranged so that each concept is closer to its descendants than to concepts in other subtrees. The blue and red dotted lines indicate the boundary of the dog and cat subtrees, respectively. Top-right: each image of a labrador is a variation of the labrador prototype, and its caption may pair with any concept on the path from the root to that leaf, so an image can be captioned labrador, dog or animal. Bottom-left: the pairing also means that the caption dog pairs with images of dog or any of its descendants, so the average of these images approaches the mean of the dog subtree. Bottom-right: contrastive learning pulls each text embedding toward the corresponding subtree mean, so the text embeddings inherit the arrangement of the image prototypes.

Datasets

The study utilizes several standard benchmarks and hierarchies:

  • ImageNet-1K: Used for the primary empirical evaluation of image embeddings.
  • WordNet: The gold-standard noun hierarchy used to define the "ground truth" taxonomy for semantic concepts.
  • CUB-200: A fine-grained bird dataset used to test the robustness of the findings across different taxonomies.
  • Synthetic Data: Custom-built hierarchies used to validate theoretical bounds in controlled settings.

Key Insights and Ideas

  • The Modality Asymmetry: One of the most striking findings is that hierarchical structure is "partially present" in raw pixels and randomly initialized image encoders. In contrast, text embeddings start as essentially random noise and only "inherit" the hierarchy through the contrastive alignment process.
  • Hierarchy as an Average: Because a general term like "dog" is paired with many specific types of dogs (Labradors, Corgis, etc.), the contrastive loss naturally pulls the text embedding for "dog" toward the mean of all dog image embeddings. If the image embeddings are already clustered hierarchically, the text embeddings will inevitably follow suit.
  • Explaining the Modality Gap: The theory suggests that optimal text embeddings align with "root-centered" image prototypes. By subtracting the global mean (the "root" of the hierarchy), the model naturally separates the text and image clusters in space, providing a geometric explanation for the well-known "modality gap" in CLIP.
  • Provable Transfer: The authors prove that under certain conditions, the percentage of hierarchical violations in the learned text embeddings is bounded by the violations already present in the image data.

Key Results

  • Empirical Verification: In pretrained CLIP models (OpenAI and OpenCLIP), HAS-Viol is remarkably low—often near 0% for deeper levels of the hierarchy. For randomly initialized text models, this violation rate is closer to 80%.
  • Depth Matters: The hierarchical structure becomes "cleaner" (lower violation rates) as you move deeper into the taxonomy.
Figure 2: Distribution of HAS-Viol vs depth on ViT-L/14 CLIP embeddings over non-root and non- leaf nodes. Top: text embeddings show that HAS is acquired through pretraining. Bottom: image embeddings show approximate HAS at random initialization and stronger HAS after pretraining.
  • Robustness: The transfer of hierarchy occurs even when the mathematical conditions for the proof (like high regularization) are relaxed, suggesting the mechanism is a fundamental property of contrastive learning on structured data.

Limitations and Future Work

  • Fixed Image Features: The theoretical proofs assume image embeddings are fixed while text embeddings are trained. While this captures the "transfer" direction, it doesn't fully model the joint dynamics where both encoders change simultaneously.
  • Origin of Image Hierarchy: The paper explains how hierarchy transfers to text, but it leaves open the deeper question of why raw image data and random filters exhibit hierarchical structure in the first place.
  • Simplified Constraints: For mathematical tractability, the authors replaced CLIP’s unit-norm constraints with $L_2$ regularization. Extending the proof to the exact normalized hypersphere remains a challenge for future work.
  • Beyond Trees: Real-world knowledge isn't always a strict tree; it's often a directed acyclic graph (DAG) or a more complex graph. Testing HAS on non-tree structures is a logical next step. Not discussed in the paper.

Summary updated

Abstract

Contrastive vision-language models such as CLIP are trained to align images and text, not to represent taxonomies. Yet empirical studies show that CLIP embeddings exhibit hierarchical structure aligned with the hierarchy of semantic concepts. A theoretical characterization of why such hierarchy emerges from a non-hierarchical objective remains open. In this paper, we formalize this phenomenon as Hierarchical Angular Separation (HAS), a geometric property whereby each non-leaf concept embedding is more similar to its descendants than to off-branch concepts. Empirically, we show that HAS holds approximately in raw images and randomly initialized CLIP image embeddings, while in text embeddings it holds approximately only after contrastive learning. To theoretically explain this asymmetry, we analyze an unconstrained features model in which the image embeddings are fixed and only the text embeddings are trained. At the global minimizer, the text-side HAS violation percentage at each node is at most the image-side HAS violation and near-violation percentage. The key mechanism is the hierarchical co-occurrence structure of image-text pairs, which biases text embeddings toward root-centered image prototypes. For a linear contrastive loss, we derive a closed-form solution that makes this mechanism explicit. We extend the analysis to the InfoNCE loss and establish analogous guarantees under sufficient regularization. Synthetic experiments validate the theory and show that HAS is transferred even beyond the regimes covered by our analysis.
BibTeX
@misc{nguyen2026provable,
  author = {Nghia Nguyen and Tianjiao Ding and Lachlan Ewen MacDonald and René Vidal},
  title = {{Provable Cross-Modal Transfer of Hierarchical Concept Structure in CLIP Embeddings}},
  year = {2026},
  eprint = {2611.17671},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17671},
}
Figure