TL;DR
This paper introduces a novel method to create word sense embeddings by integrating a semantic network's structure with existing word vector representations. This approach effectively disambiguates word meanings, leading to significant improvements in downstream NLP tasks like semantic classification.
What Problem Does This Paper Address?
Representing word meaning computationally is a cornerstone of Natural Language Processing (NLP). Traditional approaches fall into two camps: symbolic, knowledge-based methods (like using ontologies) and data-driven, distributional methods that learn word vectors from large text corpora. While distributional methods excel at capturing general word similarity, they struggle with word sense ambiguity. A single word form (like "mouse") can have multiple meanings (rodent vs. electronic device), and standard word vectors represent a conflation of all these meanings. This makes it difficult to accurately represent and utilize specific word senses, especially for less frequent ones. Existing methods for creating sense-specific vectors often rely on clustering contexts, which can be unreliable for rare senses and lack interpretability. This paper aims to bridge this gap by creating robust and interpretable sense embeddings by leveraging semantic network structures.
Methods
The core of the paper is a novel algorithm that embeds a semantic network into a continuous word space. It's based on two key intuitions:
- Sense Decomposition: Vectors for polysemous words are treated as a convex combination of their individual sense vectors. This means a word's overall vector is a weighted average of its senses, where the weights reflect the likelihood of each sense.
- Sense Neighborhood Preservation: Sense vectors are encouraged to be similar to the vectors of their neighbors in the semantic network. This leverages the relational information within the semantic network.
These intuitions are formalized into a constrained optimization problem. The authors then propose an approximate algorithm for the common case where the distance metric is squared Euclidean distance. This approximation works by iteratively adjusting sense embeddings and their "mix" (the weights from the first intuition) for each lemma, keeping neighbor embeddings fixed during each step. This online, iterative approach makes the problem computationally tractable.
Datasets
- Swedish Corpus: A 1-billion-word corpus from Språkbanken was used to generate initial lemma embeddings. This corpus was tokenized, part-of-speech tagged, and lemmatized.
- SALDO Semantic Network: The SALDO (Swedish Lexical-Semantic Network) was used as the semantic network. It contains over 125,000 entries organized by association, rather than strict is-a relationships like WordNet.
- Swedish FrameNet: A subset of 567 frames from the Swedish FrameNet was used for extrinsic evaluation. This dataset includes lexical units (LUs) which are word senses assigned to semantic frames.
Key Insights and Ideas
- Bridging Symbolic and Distributional: The method elegantly combines the strengths of symbolic knowledge (semantic networks) with distributional semantics (word vectors), creating a more nuanced representation of word meaning.
- Handling Rare Senses: By leveraging network structure, the algorithm can derive useful embeddings even for senses that appear infrequently in the corpus, overcoming a major limitation of context-clustering approaches.
- Interpretability: The "mix" variables (weights for sense combinations) provide an estimate of sense frequency, offering a baseline for word sense disambiguation and contributing to interpretability.
- Semantic Network Structure Matters: The choice of neighbors and their weights in the semantic network significantly impacts the resulting sense embeddings. The authors found that using primary descriptors and inverse primaries was effective.
Key Results
- Extrinsic Evaluation Boost: When used as features in a FrameNet lexical unit classifier, the sense embeddings resulted in a large improvement compared to using standard lemma embeddings. For example, for the "ANIMALS" frame, precision, recall, and F-measure increased from 0.741, 0.643, 0.689 to 0.826, 0.663, 0.736 respectively.
- Qualitative Inspection: Visual inspection of nearest-neighbor lists for senses of words like "rock" showed semantically coherent results, with distinct neighbors for the "coat" sense versus the "rock music" sense.
- Handling Unlisted Lemmas: The method successfully found relevant senses for lemmas present in the corpus but not explicitly listed in SALDO, demonstrating its utility beyond the annotated network.
Limitations and Future Work
- Approximation Quality: While the approximate algorithm works well in practice, its performance relative to an exact solution is not fully explored.
- Intrinsic Evaluation: The authors acknowledge that extrinsic evaluation is useful but suggest the need for intrinsic evaluation schemes, such as sense analogy tasks, to better understand the properties of the learned sense embeddings.
- Rare Sense Retrieval: The paper mentions future work on investigating the usefulness of sense vectors for retrieving rarely occurring senses in corpora.
- WSD Efficiency: A follow-up paper is cited that shows these sense embeddings can build efficient word sense disambiguation systems.
Summary updated