TL;DR
This paper tackles the problem of polysemy (words having multiple meanings) in word embeddings by proposing two novel methods to ground these embeddings in lexical ontologies like WordNet. These approaches effectively learn sense-specific word vectors, leading to significant improvements on various lexical semantic tasks compared to existing methods.
What Problem Does This Paper Address?
Traditional vector space models (VSMs) for word meaning represent each word with a single vector, failing to capture the nuances of polysemy. For instance, "bank" can refer to a financial institution or a river's edge, but a single vector struggles to represent both. While lexical ontologies like WordNet offer a structured inventory of word senses and their relationships, they haven't been effectively integrated into VSMs to address polysemy. Existing methods for handling polysemy often rely on unsupervised context clustering, which can be difficult to parametrize and results in abstract, uninterpretable vectors. This paper aims to bridge this gap by proposing methods that leverage ontologies to create sense-specific word embeddings, making them more semantically rich and interpretable.
Methods
The paper introduces two main approaches:
- Retrofitting Vectors to an Ontology: This is a post-processing technique applicable to any pre-trained, sense-agnostic VSM. It treats the ontology as a graph and uses graph smoothing to "tease apart" the vectors of different word senses. Essentially, it iteratively updates word sense vectors to be closer to their ontological neighbors while still being anchored to their original sense-agnostic word vector. This is formalized as a Markov Network inference problem.
- Adapting Predictive Models with Latent Variables and Structured Regularizers: This approach extends existing predictive models (like skip-gram) that learn word embeddings by maximizing corpus likelihood. It introduces latent variables to represent word senses and uses a structured prior based on the ontology to "ground" these sense embeddings. This method uses an Expectation-Maximization (EM) algorithm, incorporating a structured regularizer derived from the ontology to guide the learning of sense-specific vectors. The paper details how this can be applied to the skip-gram model, using a "hard" EM for efficiency.
Datasets
- Ontology: WordNet (Miller, 1995) is used as the primary lexical ontology.
- Training Data: Publicly available WMT-2011 English monolingual data (355 million words) was used for training skip-gram models.
- Evaluation Datasets:
- Similarity Scoring: WS-353, RG-65, MC-30, MEN-3k.
- Synonym Selection: ESL-50, RD-300, TOEFL-80.
- Contextual Word Similarity: SCWS.
Key Insights and Ideas
- Ontologies as a Powerful Source of Supervision: The paper demonstrates that structured knowledge from ontologies can significantly improve word embeddings by disambiguating word senses, going beyond purely distributional information.
- Interpretability of Sense Vectors: Unlike unsupervised clustering methods, the proposed techniques produce sense-specific vectors that are more interpretable and can be directly linked to ontological senses.
- Generalizability of Retrofitting: The retrofitting method is highly versatile, being applicable as a post-processing step to any existing VSM without requiring retraining from scratch.
- Trade-off between Ontology Structure and Performance: The authors note that the effectiveness of their methods is sensitive to the ontology's structure and the chosen relations, suggesting that careful tuning and more exhaustive selection procedures could further enhance performance.
- Sense-Specific Models Outperform Single-Sense Models: Consistently, the sense-specific embeddings learned by the proposed methods achieve better results on semantic tasks compared to single-sense baselines.
Key Results
- Significant Performance Gains: In most cases, the proposed sense-specific VSMs consistently and significantly outperform baseline models on similarity scoring and synonym selection tasks across multiple datasets.
- Superiority over WSD Preprocessing: The retrofitted and EM-based models perform better than approaches that preprocess corpora with unsupervised or supervised Word Sense Disambiguation (WSD) tools.
- Qualitative Analysis: Qualitative analysis shows that the sense-specific vectors effectively capture distinct meanings of polysemous words, whereas single-sense VSMs tend to represent only the most frequent sense. For example, the word "hanging" was successfully disambiguated into senses like "suspending" and "decoration," each with distinct similar words.
- Computational Efficiency: The retrofitting method is computationally efficient, scaling linearly with vocabulary size, while the EM-based approach is more computationally intensive but offers greater flexibility.
Limitations and Future Work
- Sensitivity to Ontology Structure: The performance of the models is dependent on the quality and structure of the ontology. The paper suggests that more exhaustive selection of relations and weight tuning could improve results.
- Bias Towards Similarity: The use of synonymy, hypernymy, and hyponymy relations in WordNet can favor similarity over relatedness, potentially impacting performance on datasets that require distinguishing between these.
- Future Directions:
- Using sense-specific vectors as features in downstream NLP tasks like WSD.
- Exploring a more bi-directional relationship between ontologies and VSMs, potentially involving simultaneous structure learning for both.
- Extending the research to the multi-lingual domain using multi-lingual WordNets to learn cross-lingual sense-specific semantic vectors.
Summary updated