Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts

Timo Schick, Hinrich Schütze

TL;DR

This paper introduces "attentive mimicking," a novel method for learning better word embeddings, especially for rare words. By intelligently selecting the most informative contexts for a word, their approach significantly outperforms existing methods on various downstream tasks, improving embeddings for a much larger portion of the vocabulary.

What Problem Does This Paper Address?

Learning high-quality word embeddings is crucial for many Natural Language Processing (NLP) tasks. However, standard embedding methods require a large number of word occurrences to learn a good representation. This leaves "rare" words, those that appear infrequently in text, with poor embeddings. While previous work has tried to address this by incorporating surface-form information (like character n-grams) or by using a two-step "mimicking" process (where a model learns to reproduce embeddings of frequent words), these methods still struggle with sparsity. The core challenge is that not all contexts a rare word appears in are equally useful for understanding its meaning; some contexts are noisy or irrelevant.

Methods

The paper builds upon the "form-context model" (FCM) and introduces "attentive mimicking" (AM).

  • Form-Context Model (FCM): This model first learns high-quality embeddings for frequent words. Then, for infrequent words, it combines two embeddings: one derived from the word's surface form (e.g., character n-grams) and another from averaging the embeddings of its surrounding context words. A learned weighting coefficient determines how much each of these contributes.
  • Attentive Mimicking (AM): This is the core innovation. AM enhances FCM by introducing an attention mechanism. Instead of naively averaging all context words, AM learns to assign different "reliability" weights to each context. The intuition is that reliable contexts for a word tend to agree with each other. AM formalizes this by measuring context similarity using a learned matrix and then using these similarities to compute context reliability scores. These scores are then used to create a weighted context embedding, which is combined with the surface-form embedding as in FCM.

Datasets

  • Training: The Westbury Wikipedia Corpus (WWC) was used for training all embedding models.
  • Evaluation:
    • VecMap: A novel evaluation method is introduced using a downsampled version of the WWC. This method compares learned embeddings against "gold" skipgram embeddings in a common space.
  • SemEval2015 Task 10E: Used for sentiment analysis, evaluating embeddings by predicting sentiment ratings.
  • Name Typing Dataset (Yaghoobzadeh et al., 2018): Used for fine-grained named entity type prediction.
  • Chimeras Dataset (Lazaridou et al., 2017): Consists of similarity scores for pairs of made-up and regular words, used to analyze the attention mechanism's effect on novel word representations.

Key Insights and Ideas

  • Context is not created equal: The most significant insight is that for learning embeddings of rare words, not all contexts are equally valuable. Some contexts are much more informative and reliable than others.
  • Self-attention for context selection: Applying self-attention to contexts allows the model to dynamically identify and prioritize the most relevant information, mimicking how humans might focus on specific cues.
  • Agreement as a signal of reliability: The paper proposes that contexts that "agree" with many other contexts are more likely to be reliable indicators of a word's meaning. This is a clever way to leverage the collective signal from multiple contexts.
  • Broader vocabulary improvement: AM doesn't just help the rarest words; it significantly improves embeddings for medium-frequency words too, making it beneficial for a much larger portion of the vocabulary than previous methods.

Key Results

  • VecMap Evaluation: AM consistently outperforms baselines (skipgram, fastText, Mimick) for words with fewer than 32 occurrences. AM shows improvements over FCM when given four or more contexts.
  • Sentiment Dictionary: AM outperforms all other models for rare and medium-frequency words.
  • Name Typing: AM performs drastically better than baselines for words with up to 16 occurrences. A combined approach (AM+skip) achieves the best overall results.
  • Chimeras: AM-ctx (context-only AM) shows significant improvements over FCM-ctx, especially when given more contexts (6 sentences), highlighting the benefit of attention for novel word representations.

Limitations and Future Work

  • Future Work: The authors suggest investigating whether attention mechanisms at the word level (rather than just context level) could further improve performance. They also propose exploring AM's applicability to morphologically rich languages.
  • Limitations: While AM significantly improves rare and medium-frequency words, the paper doesn't explicitly detail limitations for extremely high-frequency words where surface form might be less critical. The Chimeras dataset, while useful for analysis, is noted as not ideal for direct evaluation due to its limited contexts.

Summary updated

Abstract

Learning high-quality embeddings for rare words is a hard problem because of sparse context information. Mimicking (Pinter et al., 2017) has been proposed as a solution: given embeddings learned by a standard algorithm, a model is first trained to reproduce embeddings of frequent words from their surface form and then used to compute embeddings for rare words. In this paper, we introduce attentive mimicking: the mimicking model is given access not only to a word's surface form, but also to all available contexts and learns to attend to the most informative and reliable contexts for computing an embedding. In an evaluation on four tasks, we show that attentive mimicking outperforms previous work for both rare and medium-frequency words. Thus, compared to previous work, attentive mimicking improves embeddings for a much larger part of the vocabulary, including the medium-frequency range.
BibTeX
@inproceedings{schick2019attentive,
  author = {Timo Schick and Hinrich Schütze},
  title = {{Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=H14m-7W_ZS},
}
Figure