Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts

Timo Schick, Hinrich Schütze

Abstract

Standard word embedding techniques often fail for rare words due to sparse and noisy context. Timo Schick and Hinrich Schütze introduce Attentive Mimicking (AM), a model that improves embeddings for rare and medium-frequency words by using a self-attention mechanism to prioritize informative contexts. AM outperforms existing mimicking and form-context models across several benchmarks, including a novel intrinsic evaluation and fine-grained entity typing.

The Problem of Rare Words

Word embedding models like Word2Vec or GloVe require numerous observations to generate high-quality vector representations. This creates a "long tail" problem: the vast majority of words in a language appear infrequently, resulting in poor embeddings that degrade performance in downstream NLP tasks.

Previous solutions have attempted to address this through mimicking—predicting a rare word's embedding based on its surface form (subword n-grams)—or form-context models (FCM), which combine surface form with available context sentences. However, these models typically treat every context as equally important. In reality, many sentences containing a rare word are uninformative or noisy.

Methodology: Attentive Mimicking

Attentive Mimicking (AM) builds upon the FCM framework but introduces a mechanism to "attend" to the most useful contexts while ignoring noise. In AM, a word's embedding is a dynamic combination of its surface form and its context.

The core innovation is the self-attention mechanism used to compute context reliability. The model operates on the principle that "reliable" contexts for a specific word tend to resemble one another. If a word appears in several sentences, those that share semantic similarities are likely more informative than outliers.

  1. Similarity Scoring: AM computes similarity between contexts using a scaled dot-product attention mechanism.
  2. Weighting: Each context is assigned a reliability score ($\rho$) based on its average similarity to all other available contexts for that word.
  3. Final Embedding: The final context embedding is a weighted sum of individual context vectors based on these scores.
Reliability scores ($\rho$) assigned by AM to different context sentences for the nonce word "petfel".

As shown in the example above, the model assigns different weights to sentences based on how well they characterize the target word (in this case, the nonce word "petfel").

Evaluation and Key Results

The authors evaluated AM across several datasets, including the Westbury Wikipedia Corpus (WWC) for training and the CHIMERA dataset for "novel" words.

Name Typing

One of the most striking results came from the Name Typing task, which involves predicting fine-grained entity types (e.g., "PRESIDENT" vs. "LOCATION"). AM showed massive gains for rare words. For words appearing only once ($f=1$), AM achieved 87.8% accuracy, whereas a standard skipgram model achieved 0% because it could not learn a meaningful vector from a single observation.

Accuracy and F1 scores for the Name Typing task across different frequency bins ($f$).

Intrinsic Evaluation (VecMap)

Using a novel evaluation method called VecMap, the authors simulated rarity by downsampling frequent words. For words with only one occurrence, AM and FCM achieved cosine similarities of ~38.0 with gold embeddings, significantly outperforming the 8.7 achieved by standard skipgram.

The "Medium-Frequency" Boost

While mimicking is usually associated with rare words, AM also significantly improves embeddings for medium-frequency words (those appearing up to 32–64 times). This suggests that even with a moderate amount of data, "noise" in contexts remains a significant hurdle for standard algorithms.

Discussion and Future Work

The success of Attentive Mimicking highlights the importance of inter-context agreement as a signal for quality. By filtering out idiosyncratic usages without manual labels, AM provides a robust way to handle the long tail of language.

Future research could explore adding attention within each sentence to identify the most informative words in a specific context. Additionally, while the current evaluation focused on English, the effectiveness of the "form" component may vary in morphologically rich languages. Integrating this attentive logic directly into the training of large-scale transformer models remains a promising future direction.

Summary based on: Schick, T., & Schütze, H. (2019). Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics.

Summary updated

BibTeX
@inproceedings{schick2019attentive,
  author = {Timo Schick and Hinrich Schütze},
  title = {{Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=H14m-7W_ZS},
}
Figure