Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations

Robert Wolfe, Aylin Caliskan

Abstract

We examine the effects of contrastive visual semantic pretraining by comparing the geometry and semantic properties of contextualized English language representations formed by GPT-2 and CLIP, a zero-shot multimodal image classifier which adapts the GPT-2 architecture to encode image captions. We find that contrastive visual semantic pretraining significantly mitigates the anisotropy found in contextualized word embeddings from GPT-2, such that the intra-layer self-similarity (mean pairwise cosine similarity) of CLIP word embeddings is under .25 in all layers, compared to greater than .95 in the top layer of GPT-2. CLIP word embeddings outperform GPT-2 on word-level semantic intrinsic evaluation tasks, and achieve a new corpus-based state of the art for the RG65 evaluation, at .88. CLIP also forms fine-grained semantic representations of sentences, and obtains Spearman's rho = .73 on the SemEval-2017 Semantic Textual Similarity Benchmark with no fine-tuning, compared to no greater than rho = .45 in any layer of GPT-2. Finally, intra-layer self-similarity of CLIP sentence embeddings decreases as the layer index increases, finishing at .25 in the top layer, while the self-similarity of GPT-2 sentence embeddings formed using the EOS token increases layer-over-layer and never falls below .97. Our results indicate that high anisotropy is not an inevitable consequence of contextualization, and that visual semantic pretraining is beneficial not only for ordering visual representations, but also for encoding useful semantic representations of language, both on the word level and the sentence level.

Introduction

Modern contextualized language models (LMs) like GPT-2 are remarkably capable, yet they harbor a geometric "dirty secret": anisotropy. As information flows through the layers of these models, word embeddings tend to cluster into a narrow, high-dimensional cone. In the final layers of GPT-2, the average cosine similarity between two random words often exceeds 0.95. This suggests that the model’s internal vector space is crowded, potentially limiting its ability to distinguish between distinct semantic concepts.

This research investigates whether visual grounding—specifically the contrastive learning objective used in CLIP—can resolve this geometric collapse. By requiring a model to associate text with images, does the model develop a more robust, distributed, and semantically meaningful understanding of language?

Methodology

To ensure a controlled comparison, the researchers conducted a head-to-head analysis between GPT-2 Small and the CLIP language model. This comparison is particularly apt because CLIP utilizes a modified GPT-2 architecture as its text encoder. The primary difference lies in the training objective: GPT-2 is trained on next-token prediction (WebText), while CLIP is trained to match image-caption pairs (WebImageText) using a contrastive loss.

Experimental metrics included:

  • Intra-layer Self-Similarity: A measure of anisotropy calculated as the mean pairwise cosine similarity of embeddings within a layer.
  • Intrinsic Word Evaluation: Testing "decontextualized" word representations (single words without context) against benchmarks like RG65 and SimLex-999.
  • Sentence-Level Evaluation: Utilizing the Semantic Textual Similarity (STS) Benchmark to assess how well the models judge the relationship between sentence pairs.
  • Activation Distribution: Analyzing the magnitude of neuron activations to identify "rogue dimensions" that might dominate the representation space.

The Impact of Visual Grounding on Geometry

The most striking finding is that visual semantic pretraining acts as a powerful regularizer against anisotropy. While GPT-2's embeddings collapse into a tight cluster in deeper layers, CLIP’s word embeddings remain isotropic across the entire network, with self-similarity staying below 0.25.

This suggests that the "cone effect" is not an inherent flaw of the Transformer architecture, but rather a byproduct of the standard language modeling objective. Predicting the next word relies heavily on local syntax and statistical patterns. Conversely, matching a caption to an image requires the model to prioritize high-level semantics; the model cannot successfully link a photo of a "dog" to its caption if the vector for "dog" is nearly identical to the vector for "cat."

Semantic Performance

Word-Level Representations

CLIP demonstrated superior performance on word-level tasks, achieving a new corpus-based state-of-the-art on the RG65 evaluation with a Spearman’s $\rho$ of 0.88. This outperformed significantly larger models and more complex pooling strategies. Interestingly, the researchers discovered a "BOS trick": simply adding a [BOS] (beginning-of-sentence) token to GPT-2 improved its word-level performance, though it still trailed behind CLIP.

Sentence-Level Representations

On the STS Benchmark, CLIP achieved a Spearman’s $\rho$ of 0.73 with zero fine-tuning. In contrast, GPT-2 never surpassed 0.45 in any layer. The role of the [EOS] (end-of-sentence) token is critical here. In CLIP, the [EOS] token is the specific vector projected into the joint vision-language space, and it evolves into a highly refined semantic summary as it moves through the layers. In GPT-2, the [EOS] token becomes increasingly redundant and anisotropic, mirroring the collapse seen in word embeddings.

Distributed Neural Capacity

Analysis of neuron activations revealed that GPT-2's representations are dominated by a few "rogue dimensions." In its top layer, just five neurons account for 97% of the embedding's magnitude. In CLIP, the top five neurons account for only 39%, indicating that visual pretraining encourages the model to use its full capacity more efficiently, distributing information across the vector space.

Conclusion and Future Directions

These results provide compelling evidence that visual semantic pretraining fundamentally improves the quality of natural language representations. By grounding text in the visual world, models move beyond syntactic mimicry toward a more robust semantic understanding.

However, questions remain regarding whether these benefits stem from the visual nature of the data or the contrastive nature of the objective. Future research should compare CLIP against models trained with contrastive objectives on text alone. Additionally, while CLIP's representations are semantically magnified, they also risk capturing and amplifying societal biases present in large-scale internet datasets. Ultimately, this work suggests that for AI to truly understand language, it may need to look at the world, not just read about it.

Summary updated

Abstract

We examine the effects of contrastive visual semantic pretraining by comparing the geometry and semantic properties of contextualized English language representations formed by GPT-2 and CLIP, a zero-shot multimodal image classifier which adapts the GPT-2 architecture to encode image captions. We find that contrastive visual semantic pretraining significantly mitigates the anisotropy found in contextualized word embeddings from GPT-2, such that the intra-layer self-similarity (mean pairwise cosine similarity) of CLIP word embeddings is under .25 in all layers, compared to greater than .95 in the top layer of GPT-2. CLIP word embeddings outperform GPT-2 on word-level semantic intrinsic evaluation tasks, and achieve a new corpus-based state of the art for the RG65 evaluation, at .88. CLIP also forms fine-grained semantic representations of sentences, and obtains Spearman's rho = .73 on the SemEval-2017 Semantic Textual Similarity Benchmark with no fine-tuning, compared to no greater than rho = .45 in any layer of GPT-2. Finally, intra-layer self-similarity of CLIP sentence embeddings decreases as the layer index increases, finishing at .25 in the top layer, while the self-similarity of GPT-2 sentence embeddings formed using the EOS token increases layer-over-layer and never falls below .97. Our results indicate that high anisotropy is not an inevitable consequence of contextualization, and that visual semantic pretraining is beneficial not only for ordering visual representations, but also for encoding useful semantic representations of language, both on the word level and the sentence level.
BibTeX
@inproceedings{wolfe2022contrastive,
  author = {Robert Wolfe and Aylin Caliskan},
  title = {{Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=4aqpgsSkVlJ},
}
Figure