TL;DR
This paper tackles the "representation degeneration" problem in contextual word embeddings, where anisotropy (a lack of uniform distribution) hinders their ability to capture distinct word senses. They propose LASER, a post-processing method that first reduces anisotropy and then retrofits embeddings to improve sense disambiguation, leading to more semantically meaningful representations.
What Problem Does This Paper Address?
Modern NLP models like BERT and GPT-2 generate contextual word representations, meaning a word can have different embeddings depending on its usage. This is a significant improvement over static embeddings (like word2vec) which conflate multiple meanings. However, researchers have observed that these contextual representations often exhibit anisotropy. Geometrically, this means the word vectors are not uniformly distributed in space but tend to cluster in narrow cones. This anisotropy is problematic because it can mask the true semantic relationships between words and, crucially, obscure the model's ability to distinguish between different senses of the same word. The paper argues that this "representation degeneration" limits the utility of these powerful models as knowledge resources, and they aim to address this by making representations more isotropic and sense-enriched.
Methods
The paper introduces LASER (Low Anisotropy Sense Retrofitting), a two-stage post-processing approach:
- Low Anisotropy: This stage aims to reduce the anisotropy of the embeddings. It builds on prior work by using Principal Component Analysis (PCA) to identify and remove the dominant directions in the vector space. The idea is that by removing these common directions, the remaining vectors become more uniformly distributed.
- Sense Retrofitting: This stage enhances the semantic meaning by enforcing lexical constraints. It's an extension of traditional retrofitting methods, which typically align word vectors with semantic lexicons. LASER adapts this to contextual embeddings by aiming to pull representations of the same word sense closer together while pushing representations of different word senses farther apart. This is achieved through an update rule that considers neighboring word representations within a knowledge graph.
Datasets
The authors used sense-annotated corpora from various SemEval and SenseEval tasks to evaluate sense learning. Specifically, they mention:
- SensEval 3 task 1 (S3-T1)
- SensEval 2 all-words task (S2-TA)
- SemEval 2013 task 12 (S13-T12)
- SemEval 2007 task 7 (S7-T7)
- SemEval 2015 task 13 (S15-T13)
They unified WordNet 3.0 sense keys across these corpora. The analysis focused on multi-sense words (nouns, adjectives, and verbs).
Key Insights and Ideas
- Anisotropy is pervasive: All tested language models (BERT, GPT-2, XLNet, ELECTRA) exhibit anisotropy across their layers.
- Unidirectional vs. Bidirectional: Unidirectional models (GPT-2, XLNet) tend to be more anisotropic than bidirectional models (BERT, ELECTRA). This suggests that learning context from all directions helps create more uniform representations.
- Layer-wise anisotropy: Anisotropy generally increases in the upper layers of models trained on masked language modeling tasks, indicating that contextualization can exacerbate the degeneration problem.
- Sense information is masked by anisotropy: The paper argues that without accounting for anisotropy, conclusions about sense learning capabilities can be misleading.
- LASER effectively reduces anisotropy: The proposed method significantly improves isotropy, making representations more uniformly distributed.
- LASER enhances sense disambiguation: After retrofitting, representations of the same word sense become more cohesive, and representations of different senses become more separated, leading to better sense discrimination.
Key Results
- Isotropy Improvement: After applying LASER's anisotropy reduction step, the average cosine similarity between randomly sampled words approaches 0, indicating a significant reduction in anisotropy.
- Sense Similarity Boost: Table 2 shows that LASER consistently increases the
SenseSim(average sense similarity) scores across all models and layers. For instance, BERT'sSenseSimincreases from around 0.2-0.8 to 0.6-0.93 after retrofitting. - Improved Sense Distinction (Δ): The
Δmetric (SenseSim - InterSenseSim) also shows significant improvement after LASER, indicating better separation between different senses. For example, GPT-2'sΔjumps from near 0 to around 0.38-0.43. - Qualitative Visualization: Figures 1, 8, 9, 10, and 11 visually demonstrate how LASER transforms the PCA plots of word representations, making them less clustered and more spread out, especially for highly anisotropic models like GPT-2.
Limitations and Future Work
- Hyperparameter Tuning: The paper mentions that the impact of changes in retrofitting hyperparameters and the number of principal components removed needs further study.
- Extrinsic Evaluation: The authors plan to conduct extrinsic evaluations to assess the impact of anisotropy removal and sense retrofitting on downstream word-sense disambiguation tasks.
- Unseen Words: Future work will focus on extending LASER to handle words that were not seen during the retrofitting process.
Summary updated