Abstract
Standard language models trained on text-only objectives often suffer from anisotropy, where embeddings occupy a narrow cone in vector space, limiting semantic differentiation. We demonstrate that grounding language in a visual-semantic space via contrastive pretraining (as in CLIP) fundamentally alters this geometry. Our analysis shows that CLIP's language representations are significantly more isotropic and semantically rich than those of GPT-2, achieving state-of-the-art performance on word and sentence similarity benchmarks without fine-tuning.
Introduction
Modern Large Language Models (LLMs) like GPT-2 exhibit a geometric phenomenon known as anisotropy. In high-dimensional space, their word embeddings are highly correlated, clustering into a narrow "cone." This lack of spread makes it difficult for models to distinguish fine-grained semantic nuances. While previously considered an inherent byproduct of contextualization, this research investigates whether anisotropy is instead a consequence of the training objective. We explore if grounding language in the visual world—forcing a model to distinguish between diverse visual scenes—can promote more distinct and semantically meaningful representations.
Methods
We perform a comparative analysis between two models sharing the same Transformer architecture (GPT-2 Small) but trained under different paradigms:
- GPT-2 Small: Trained on the WebText corpus using a standard causal language modeling objective (next-token prediction).
- CLIP (Language Encoder): Trained on WebImageText (400 million pairs) using a contrastive visual-semantic objective, matching text captions to corresponding images.
To quantify the geometric properties, we measure intra-layer self-similarity (average cosine similarity between random embedding pairs) using 36,000 sentences from SemEval STS tasks. We further evaluate semantic quality using five word-level benchmarks (RG65, WordSim-353, SimLex-999, SimVerb-3500, ValNorm) and the SemEval-2017 STS sentence-level benchmark.
Results
Geometric Transformation and Isotropy
The most significant finding is that CLIP’s embeddings are nearly isotropic, effectively utilizing the available vector space. While GPT-2's top-layer word embeddings reach a self-similarity of over 0.95, CLIP’s remain below 0.25. This indicates that the "narrow cone" is not a structural limitation of Transformers but a result of the training objective.
Furthermore, the distribution of information across neurons is more efficient in CLIP. In GPT-2's final layer, the five most active neurons account for over 97% of an embedding's magnitude. In contrast, the top five neurons in CLIP account for only 39%, suggesting a more distributed and robust representation.
Semantic Performance
Visual pretraining leads to superior performance on intrinsic semantic tasks. CLIP achieved a new state-of-the-art score on the RG65 word similarity task (Spearman’s $\rho = .88$). On sentence-level tasks (STS Benchmark), CLIP reached $\rho = .73$ with zero fine-tuning, significantly outperforming GPT-2, which peaked at $\rho = .45$.
A key driver of this performance in CLIP is the End of Sentence (EOS) token. During training, this token is projected into the joint image-text space, causing it to aggregate the semantic essence of the entire sentence.
Discussion
Our results suggest that visual grounding forces the model to differentiate concepts more sharply to succeed at the contrastive task. Unlike next-token prediction, which can rely on local syntax, matching a caption to a specific image requires global semantic distinctness. This "magnified semantics" extends beyond visual terms to abstract concepts. While it remains to be seen how much of this benefit stems from the contrastive algorithm versus the visual data itself, it is clear that visual-semantic pretraining provides a powerful mechanism for overcoming anisotropy and enriching natural language representations at the source.
Summary updated