Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations

Geetanjali Bihani, Julia RayzNAACL 2021

TL;DR

This paper tackles the "representation degeneration" problem in contextual word embeddings, where anisotropy (a lack of uniform distribution) hinders their ability to capture distinct word senses. They propose LASER, a post-processing method that first reduces anisotropy and then retrofits embeddings to improve sense disambiguation, leading to more semantically meaningful representations.

What Problem Does This Paper Address?

Modern NLP models like BERT and GPT-2 generate contextual word representations, meaning a word can have different embeddings depending on its usage. This is a significant improvement over static embeddings (like word2vec) which conflate multiple meanings. However, researchers have observed that these contextual representations often exhibit anisotropy. Geometrically, this means the word vectors are not uniformly distributed in space but tend to cluster in narrow cones. This anisotropy is problematic because it can mask the true semantic relationships between words and, crucially, obscure the model's ability to distinguish between different senses of the same word. The paper argues that this "representation degeneration" limits the utility of these powerful models as knowledge resources, and they aim to address this by making representations more isotropic and sense-enriched.

Methods

The paper introduces LASER (Low Anisotropy Sense Retrofitting), a two-stage post-processing approach:

  1. Low Anisotropy: This stage aims to reduce the anisotropy of the embeddings. It builds on prior work by using Principal Component Analysis (PCA) to identify and remove the dominant directions in the vector space. The idea is that by removing these common directions, the remaining vectors become more uniformly distributed.
  2. Sense Retrofitting: This stage enhances the semantic meaning by enforcing lexical constraints. It's an extension of traditional retrofitting methods, which typically align word vectors with semantic lexicons. LASER adapts this to contextual embeddings by aiming to pull representations of the same word sense closer together while pushing representations of different word senses farther apart. This is achieved through an update rule that considers neighboring word representations within a knowledge graph.

Datasets

The authors used sense-annotated corpora from various SemEval and SenseEval tasks to evaluate sense learning. Specifically, they mention:

  • SensEval 3 task 1 (S3-T1)
  • SensEval 2 all-words task (S2-TA)
  • SemEval 2013 task 12 (S13-T12)
  • SemEval 2007 task 7 (S7-T7)
  • SemEval 2015 task 13 (S15-T13)

They unified WordNet 3.0 sense keys across these corpora. The analysis focused on multi-sense words (nouns, adjectives, and verbs).

Key Insights and Ideas

  • Anisotropy is pervasive: All tested language models (BERT, GPT-2, XLNet, ELECTRA) exhibit anisotropy across their layers.
  • Unidirectional vs. Bidirectional: Unidirectional models (GPT-2, XLNet) tend to be more anisotropic than bidirectional models (BERT, ELECTRA). This suggests that learning context from all directions helps create more uniform representations.
  • Layer-wise anisotropy: Anisotropy generally increases in the upper layers of models trained on masked language modeling tasks, indicating that contextualization can exacerbate the degeneration problem.
  • Sense information is masked by anisotropy: The paper argues that without accounting for anisotropy, conclusions about sense learning capabilities can be misleading.
  • LASER effectively reduces anisotropy: The proposed method significantly improves isotropy, making representations more uniformly distributed.
  • LASER enhances sense disambiguation: After retrofitting, representations of the same word sense become more cohesive, and representations of different senses become more separated, leading to better sense discrimination.

Key Results

  • Isotropy Improvement: After applying LASER's anisotropy reduction step, the average cosine similarity between randomly sampled words approaches 0, indicating a significant reduction in anisotropy.
  • Sense Similarity Boost: Table 2 shows that LASER consistently increases the SenseSim (average sense similarity) scores across all models and layers. For instance, BERT's SenseSim increases from around 0.2-0.8 to 0.6-0.93 after retrofitting.
  • Improved Sense Distinction (Δ): The Δ metric (SenseSim - InterSenseSim) also shows significant improvement after LASER, indicating better separation between different senses. For example, GPT-2's Δ jumps from near 0 to around 0.38-0.43.
  • Qualitative Visualization: Figures 1, 8, 9, 10, and 11 visually demonstrate how LASER transforms the PCA plots of word representations, making them less clustered and more spread out, especially for highly anisotropic models like GPT-2.

Limitations and Future Work

  • Hyperparameter Tuning: The paper mentions that the impact of changes in retrofitting hyperparameters and the number of principal components removed needs further study.
  • Extrinsic Evaluation: The authors plan to conduct extrinsic evaluations to assess the impact of anisotropy removal and sense retrofitting on downstream word-sense disambiguation tasks.
  • Unseen Words: Future work will focus on extending LASER to handle words that were not seen during the retrofitting process.

Summary updated

BibTeX
@inproceedings{bihani2021low,
  author = {Geetanjali Bihani and Julia Rayz},
  title = {{Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure