Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations

Robert Wolfe, Aylin Caliskan

Abstract

Standard language models trained on text-only objectives often suffer from anisotropy, where embeddings occupy a narrow cone in vector space, limiting semantic differentiation. We demonstrate that grounding language in a visual-semantic space via contrastive pretraining (as in CLIP) fundamentally alters this geometry. Our analysis shows that CLIP's language representations are significantly more isotropic and semantically rich than those of GPT-2, achieving state-of-the-art performance on word and sentence similarity benchmarks without fine-tuning.

Introduction

Modern Large Language Models (LLMs) like GPT-2 exhibit a geometric phenomenon known as anisotropy. In high-dimensional space, their word embeddings are highly correlated, clustering into a narrow "cone." This lack of spread makes it difficult for models to distinguish fine-grained semantic nuances. While previously considered an inherent byproduct of contextualization, this research investigates whether anisotropy is instead a consequence of the training objective. We explore if grounding language in the visual world—forcing a model to distinguish between diverse visual scenes—can promote more distinct and semantically meaningful representations.

Methods

We perform a comparative analysis between two models sharing the same Transformer architecture (GPT-2 Small) but trained under different paradigms:

  1. GPT-2 Small: Trained on the WebText corpus using a standard causal language modeling objective (next-token prediction).
  2. CLIP (Language Encoder): Trained on WebImageText (400 million pairs) using a contrastive visual-semantic objective, matching text captions to corresponding images.

To quantify the geometric properties, we measure intra-layer self-similarity (average cosine similarity between random embedding pairs) using 36,000 sentences from SemEval STS tasks. We further evaluate semantic quality using five word-level benchmarks (RG65, WordSim-353, SimLex-999, SimVerb-3500, ValNorm) and the SemEval-2017 STS sentence-level benchmark.

Results

Geometric Transformation and Isotropy

The most significant finding is that CLIP’s embeddings are nearly isotropic, effectively utilizing the available vector space. While GPT-2's top-layer word embeddings reach a self-similarity of over 0.95, CLIP’s remain below 0.25. This indicates that the "narrow cone" is not a structural limitation of Transformers but a result of the training objective.

CWE Self-Similarity by Layer: GPT-2 shows high anisotropy (high similarity) in deeper layers, while CLIP remains isotropic.

Furthermore, the distribution of information across neurons is more efficient in CLIP. In GPT-2's final layer, the five most active neurons account for over 97% of an embedding's magnitude. In contrast, the top five neurons in CLIP account for only 39%, suggesting a more distributed and robust representation.

Semantic Performance

Visual pretraining leads to superior performance on intrinsic semantic tasks. CLIP achieved a new state-of-the-art score on the RG65 word similarity task (Spearman’s $\rho = .88$). On sentence-level tasks (STS Benchmark), CLIP reached $\rho = .73$ with zero fine-tuning, significantly outperforming GPT-2, which peaked at $\rho = .45$.

Performance by Intrinsic Evaluation Task: CLIP consistently outperforms GPT-2 variants across multiple word and sentence similarity benchmarks.

A key driver of this performance in CLIP is the End of Sentence (EOS) token. During training, this token is projected into the joint image-text space, causing it to aggregate the semantic essence of the entire sentence.

Discussion

Our results suggest that visual grounding forces the model to differentiate concepts more sharply to succeed at the contrastive task. Unlike next-token prediction, which can rely on local syntax, matching a caption to a specific image requires global semantic distinctness. This "magnified semantics" extends beyond visual terms to abstract concepts. While it remains to be seen how much of this benefit stems from the contrastive algorithm versus the visual data itself, it is clear that visual-semantic pretraining provides a powerful mechanism for overcoming anisotropy and enriching natural language representations at the source.

Summary updated

Abstract

We examine the effects of contrastive visual semantic pretraining by comparing the geometry and semantic properties of contextualized English language representations formed by GPT-2 and CLIP, a zero-shot multimodal image classifier which adapts the GPT-2 architecture to encode image captions. We find that contrastive visual semantic pretraining significantly mitigates the anisotropy found in contextualized word embeddings from GPT-2, such that the intra-layer self-similarity (mean pairwise cosine similarity) of CLIP word embeddings is under .25 in all layers, compared to greater than .95 in the top layer of GPT-2. CLIP word embeddings outperform GPT-2 on word-level semantic intrinsic evaluation tasks, and achieve a new corpus-based state of the art for the RG65 evaluation, at .88. CLIP also forms fine-grained semantic representations of sentences, and obtains Spearman's rho = .73 on the SemEval-2017 Semantic Textual Similarity Benchmark with no fine-tuning, compared to no greater than rho = .45 in any layer of GPT-2. Finally, intra-layer self-similarity of CLIP sentence embeddings decreases as the layer index increases, finishing at .25 in the top layer, while the self-similarity of GPT-2 sentence embeddings formed using the EOS token increases layer-over-layer and never falls below .97. Our results indicate that high anisotropy is not an inevitable consequence of contextualization, and that visual semantic pretraining is beneficial not only for ordering visual representations, but also for encoding useful semantic representations of language, both on the word level and the sentence level.
BibTeX
@inproceedings{wolfe2022contrastive,
  author = {Robert Wolfe and Aylin Caliskan},
  title = {{Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=4aqpgsSkVlJ},
}
Figure