TL;DR
This paper introduces "attentive mimicking," a novel method for learning better word embeddings, especially for rare words. By intelligently selecting the most informative contexts for a word, their approach significantly outperforms existing methods on various downstream tasks, improving embeddings for a much larger portion of the vocabulary.
What Problem Does This Paper Address?
Learning high-quality word embeddings is crucial for many Natural Language Processing (NLP) tasks. However, standard embedding methods require a large number of word occurrences to learn a good representation. This leaves "rare" words, those that appear infrequently in text, with poor embeddings. While previous work has tried to address this by incorporating surface-form information (like character n-grams) or by using a two-step "mimicking" process (where a model learns to reproduce embeddings of frequent words), these methods still struggle with sparsity. The core challenge is that not all contexts a rare word appears in are equally useful for understanding its meaning; some contexts are noisy or irrelevant.
Methods
The paper builds upon the "form-context model" (FCM) and introduces "attentive mimicking" (AM).
- Form-Context Model (FCM): This model first learns high-quality embeddings for frequent words. Then, for infrequent words, it combines two embeddings: one derived from the word's surface form (e.g., character n-grams) and another from averaging the embeddings of its surrounding context words. A learned weighting coefficient determines how much each of these contributes.
- Attentive Mimicking (AM): This is the core innovation. AM enhances FCM by introducing an attention mechanism. Instead of naively averaging all context words, AM learns to assign different "reliability" weights to each context. The intuition is that reliable contexts for a word tend to agree with each other. AM formalizes this by measuring context similarity using a learned matrix and then using these similarities to compute context reliability scores. These scores are then used to create a weighted context embedding, which is combined with the surface-form embedding as in FCM.
Datasets
- Training: The Westbury Wikipedia Corpus (WWC) was used for training all embedding models.
- Evaluation:
- VecMap: A novel evaluation method is introduced using a downsampled version of the WWC. This method compares learned embeddings against "gold" skipgram embeddings in a common space.
- SemEval2015 Task 10E: Used for sentiment analysis, evaluating embeddings by predicting sentiment ratings.
- Name Typing Dataset (Yaghoobzadeh et al., 2018): Used for fine-grained named entity type prediction.
- Chimeras Dataset (Lazaridou et al., 2017): Consists of similarity scores for pairs of made-up and regular words, used to analyze the attention mechanism's effect on novel word representations.
Key Insights and Ideas
- Context is not created equal: The most significant insight is that for learning embeddings of rare words, not all contexts are equally valuable. Some contexts are much more informative and reliable than others.
- Self-attention for context selection: Applying self-attention to contexts allows the model to dynamically identify and prioritize the most relevant information, mimicking how humans might focus on specific cues.
- Agreement as a signal of reliability: The paper proposes that contexts that "agree" with many other contexts are more likely to be reliable indicators of a word's meaning. This is a clever way to leverage the collective signal from multiple contexts.
- Broader vocabulary improvement: AM doesn't just help the rarest words; it significantly improves embeddings for medium-frequency words too, making it beneficial for a much larger portion of the vocabulary than previous methods.
Key Results
- VecMap Evaluation: AM consistently outperforms baselines (skipgram, fastText, Mimick) for words with fewer than 32 occurrences. AM shows improvements over FCM when given four or more contexts.
- Sentiment Dictionary: AM outperforms all other models for rare and medium-frequency words.
- Name Typing: AM performs drastically better than baselines for words with up to 16 occurrences. A combined approach (AM+skip) achieves the best overall results.
- Chimeras: AM-ctx (context-only AM) shows significant improvements over FCM-ctx, especially when given more contexts (6 sentences), highlighting the benefit of attention for novel word representations.
Limitations and Future Work
- Future Work: The authors suggest investigating whether attention mechanisms at the word level (rather than just context level) could further improve performance. They also propose exploring AM's applicability to morphologically rich languages.
- Limitations: While AM significantly improves rare and medium-frequency words, the paper doesn't explicitly detail limitations for extremely high-frequency words where surface form might be less critical. The Chimeras dataset, while useful for analysis, is noted as not ideal for direct evaluation due to its limited contexts.
Summary updated