How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings

Kawin EthayarajhEMNLP 2019 · OpenReview

Abstract

Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks. However, just how contextual are the contextualized representations produced by models such as ELMo and BERT? Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations? For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers. This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations. In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.

Introduction

The transition from static word embeddings, such as Word2Vec and GloVe, to contextualized representations like ELMo, BERT, and GPT-2 has revolutionized natural language processing. While static embeddings assign a single vector to a word regardless of its usage, contextualized models generate representations that vary based on the surrounding text. However, the precise nature of this contextualization remained poorly understood. This study investigates whether these models represent words through a finite set of senses or truly unique, context-specific vectors, and how the geometry of these vector spaces evolves across model layers.

Methodology

To quantify contextuality, the study introduces three metrics, each adjusted for anisotropy—the tendency of word vectors to cluster in a narrow cone rather than occupying the space uniformly:

  1. Self-Similarity: The average cosine similarity of a word across different contexts. Lower similarity indicates higher contextuality.
  2. Intra-Sentence Similarity: The average cosine similarity between a word and others in the same sentence.
  3. Maximum Explainable Variance (MEV): A PCA-based measure of how much variation in contextualized representations can be captured by a single static vector.

The analysis was performed on ELMo, BERT, and GPT-2 using the SemEval Semantic Textual Similarity (STS) datasets (2012–2016), which provide diverse contexts for identical words.

The Geometry of Contextualized Space

A primary finding is the "Narrow Cone" phenomenon, or anisotropy. In most layers, word vectors are not distributed isotropically but are concentrated in a limited direction. This effect is most pronounced in GPT-2; by its final layer, the similarity between two random, unrelated words approaches 1.0, suggesting the model uses a very small portion of the available vector space.

Figure 1: Average cosine similarity between randomly sampled words across layers, illustrating the anisotropy or "narrow cone" effect.

Layer-wise Contextuality

The study finds that contextuality generally increases with model depth. In the lower layers, representations behave more like static embeddings. As one moves to upper layers, self-similarity decreases significantly, indicating that representations become increasingly specific to their immediate context. Interestingly, functional "stopwords" (e.g., "the", "of") exhibit the highest contextuality, as their meaning is almost entirely derived from their grammatical role in a specific sentence.

Figure 2: Anisotropy-adjusted average self-similarity across layers. Lower similarity indicates higher contextuality in upper layers.

Key Results and Implications

The research demonstrates that static embeddings are fundamentally insufficient for capturing the nuance of modern models. On average, less than 5% of the variance in a word's contextualized representations can be explained by a single static vector. This provides a mathematical justification for why contextualized models consistently outperform static ones.

However, the study also reveals that high-quality static embeddings can be derived from these models. By extracting the first principal component from lower layers (notably BERT Layer 1), the author produced static embeddings that outperform traditional benchmarks like GloVe and FastText on word-vector tasks.

Conclusion

This geometric analysis confirms that contextualized models produce highly specific representations that occupy a narrow, anisotropic subspace. While the study highlights the superiority of these representations over static ones, it also raises questions about the origins of anisotropy—whether it stems from model architecture or training objectives—and suggests that future training could benefit from explicitly penalizing this "narrow cone" effect to improve vector space efficiency.

Summary updated

Abstract

Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks. However, just how contextual are the contextualized representations produced by models such as ELMo and BERT? Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations? For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers. This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations. In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.
BibTeX
@inproceedings{ethayarajh2019how,
  author = {Kawin Ethayarajh},
  title = {{How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=NC7eVxB8AbZ},
}
Figure