A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space

Sara Rajaee, Mohammad Taher PilehvarACL 2021

TL;DR

This paper tackles the "representation degeneration" problem in contextual word embeddings, where word vectors become overly similar, hindering their expressiveness. They propose a novel cluster-based method that locally removes dominant directions within specific word clusters, significantly improving isotropy and downstream task performance compared to global approaches.

What Problem Does This Paper Address?

Contextual Word Representations (CWRs), like those from BERT and GPT-2, are powerful but suffer from a phenomenon called "representation degeneration." This leads to an "anisotropic" embedding space, meaning word vectors tend to cluster in a narrow cone. In such a space, even unrelated words can have high cosine similarity, making the embeddings less expressive and limiting the diversity of encoded knowledge. Existing methods to fix this often involve global retraining or analysis, which might miss the nuances of CWRs' inherently clustered structure. The authors argue that a local, cluster-aware approach is needed to effectively address this anisotropy.

Methods

The core of the paper is a cluster-based approach for improving isotropy.

  1. Clustering: They first cluster word embeddings using a standard k-means algorithm.
  2. Local Dominant Direction Identification: For each cluster, they apply Principal Component Analysis (PCA) to identify the principal components (PCs). These PCs represent the directions of maximum variance within that specific cluster, essentially capturing "dominant directions."
  3. Dominant Direction Removal: The key innovation is to nullify (remove) these cluster-specific dominant directions. This is done locally, meaning only the dominant directions for a particular cluster are removed, rather than globally.
  4. Zero-Mean Operation: Before PCA, each cluster is made zero-mean.

This approach is contrasted with a "global approach" that identifies and removes dominant directions from the entire embedding space.

Datasets

The paper uses several standard datasets for evaluation:

  • Semantic Textual Similarity (STS): STS 2012-2016, SICK-R, and STS-B.
  • Classification Tasks: GLUE benchmark (RTE, CoLA, SST-2), MRPC, WiC, and BoolQ.

The authors also mention using a dataset created by Ravfogel et al. (2020) for analyzing structural and syntactic information in punctuation and stop word clusters, and SemCor (Miller et al., 1993) for analyzing verb tense and meaning.

Key Insights and Ideas

  • Local vs. Global Isotropy: The paper strongly advocates for a local, cluster-based assessment and improvement of isotropy, arguing that global measures can be misleading due to the inherent clustered nature of CWRs.
  • Dominant Directions Encode Structural/Tense Information: The dominant directions identified in clusters are not random noise. They encode specific linguistic information:
  • Punctuation and Stop Words: Dominant directions in these clusters often capture structural and syntactic information about sentence style, rather than pure semantics. Removing them can reduce this "bias."
    • Verbs: Dominant directions in verb clusters are heavily influenced by tense. Removing them helps align representations of the same verb sense across different tenses.
  • Cosine Similarity as a Potentially Misleading Metric: The paper points out that while cosine similarity is often used to measure isotropy, it can be misleading in anisotropic spaces where clusters are aligned, leading to near-zero cosine similarity for random pairs even when the space is far from isotropic.

Key Results

  • Significant Isotropy Improvement: The cluster-based approach consistently and significantly increases the isotropy of embedding spaces across multiple STS datasets, as measured by the I(W) metric. For instance, on STS-B, the cluster-based method achieves isotropy values of 0.71-0.93 for different models, compared to baseline values as low as 1.4E-178.
  • Performance Gains on Downstream Tasks: The improved isotropy translates to substantial performance improvements on various tasks. For example, on STS tasks, the cluster-based approach yields higher Spearman correlation scores than both the baseline and the global approach. Table 2 shows improvements across multiple STS datasets, with the cluster-based method outperforming the global approach for all models. Table 3 shows similar gains on classification tasks.
  • Reduced Convergence Time: Figure 4 demonstrates that by reducing the correlation between embeddings, the cluster-based method can lead to faster convergence times for deep neural networks trained on these embeddings.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that even after their approach, some embedding spaces (like GPT-2) are still far from being perfectly isotropic, suggesting that more complex structures might be at play.
  • The paper notes that the dominant directions in some clusters might relate to word frequency bias, which their method helps to mitigate but doesn't entirely eliminate.
  • Future Work:
    • The authors plan to investigate the effect of fine-tuning on isotropy and the encoded linguistic knowledge in local regions.
  • Further exploration of how to best leverage or interpret the linguistic knowledge encoded in dominant directions.

Summary updated

BibTeX
@inproceedings{rajaee2021cluster,
  author = {Sara Rajaee and Mohammad Taher Pilehvar},
  title = {{A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure