Abstract
Word embeddings play a significant role in many modern NLP systems. However, most used word embedding learning methods learn one representation per word which is problematic for polysemous words and homonymous words. To address this problem, we propose a multi-phase word sense embedding retrofitting method which utilizes a lexical ontology to learn one embedding per word sense. We use word sense definitions and relations between word senses defined in a lexical ontology in a different way from existing systems. Experimental results on word similarity task show that our approach remarkablely improves the quality of embeddings.
Introduction
Standard word embeddings like Word2Vec and GloVe suffer from a "one-word-one-vector" problem, failing to distinguish between different meanings of the same word. For instance, the word "bank" can refer to a river edge or a financial institution. This is problematic for polysemous words (one word, multiple related meanings) and homonymous words (one word, multiple unrelated meanings). While previous researchers have tried to solve this by inducing senses from unannotated text, those methods are often noisy or fail to utilize the rich, structured data found in lexical ontologies like WordNet.
As shown in the figure above, a word like "adhere" can have several distinct senses ranging from physical sticking to being a loyal follower. The authors aim to bridge this gap by using both the definitions (glosses) and the relational structure (like "is-a" or "part-of" relationships) of an ontology to refine existing embeddings.
Methodology
The authors propose a five-phase "retrofitting" pipeline to move from word-level vectors to sense-level vectors:
- Embedding Initialization: Start with pre-trained vectors (Skip-gram or GloVe). Every sense of a word is initially assigned that word's generic vector.
- RNN for Definition Understanding: They use a Recurrent Neural Network (specifically comparing standard RNN, GRU, and LSTM) to process the dictionary definitions of words. They first train this on monosemes to teach the RNN how to map a text definition to a vector space.
- Joint Sense Learning: They then update all sense embeddings and the RNN parameters together, using the definitions to "pull" the sense vectors toward their descriptive meanings.
- Relation Mapping: WordNet contains relations like hypernymy. The authors train specific mapping matrices for each relation type, ensuring that related senses maintain a structured mathematical relationship.
- Joint Training: A final phase where all parameters—sense embeddings, RNN weights, and relation matrices—are fine-tuned together.
Results and Discussion
The model was evaluated on the Stanford’s Contextual Word Similarities (SCWS) task, which provides context for a word to allow for sense selection before similarity calculation. The results demonstrate that adding definitions provides the biggest performance jump, while adding semantic relations provides a smaller, but consistent, additional boost.
As seen in the results table, the Gated Recurrent Unit (GRU) consistently outperformed LSTMs and standard RNNs for definition understanding. The multi-phase approach acts as a careful initialization sequence, preventing the "poisoning" of the high-quality pre-trained embedding space that might occur with random parameter initialization in joint training.
Conclusion
This research highlights that definitions contain the "essential information" of a sense and that an RNN can effectively translate this linguistic description into a mathematical coordinate. By using mapping matrices to model relations, the approach acknowledges that different semantic relationships (like "part-of" vs "antonym") are qualitatively different. While the method relies on high-quality lexical ontologies, it significantly improves embedding quality for polysemous words. Future work could explore how these sense-specific vectors perform in downstream tasks like machine translation or sentiment analysis.
Summary updated