Rare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive Mimicking

Timo Schick, Hinrich Schütze

Abstract

Pretraining deep neural network architectures with a language modeling objective has brought large improvements for many natural language processing tasks. Exemplified by BERT, a recently proposed such architecture, we demonstrate that despite being trained on huge amounts of data, deep language models still struggle to understand rare words. To fix this problem, we adapt Attentive Mimicking, a method that was designed to explicitly learn embeddings for rare words, to deep language models. In order to make this possible, we introduce one-token approximation, a procedure that enables us to use Attentive Mimicking even when the underlying language model uses subword-based tokenization, i.e., it does not assign embeddings to all words. To evaluate our method, we create a novel dataset that tests the ability of language models to capture semantic properties of words without any task-specific fine-tuning. Using this dataset, we show that adding our adapted version of Attentive Mimicking to BERT does substantially improve its understanding of rare words.

Introduction: The Rare Word Problem

While models like BERT have revolutionized NLP, they suffer from a significant "rare word" problem. To handle large vocabularies efficiently, these models employ subword tokenization (e.g., WordPiece or BPE). Consequently, a rare word like "kumquat" is not treated as a single semantic entity but is instead fragmented into pieces like kum + ##quat.

The authors demonstrate that even when trained on massive datasets, BERT fails to capture the semantic properties of these infrequent terms. While the model correctly identifies a "lime" as a fruit, it may erroneously guess that a "kumquat" is a "horse" or a "dog." This failure stems from the lack of a high-quality, dedicated embedding for the full word, forcing the model to rely on the generic meanings of its fragments.

Examples of BERT's failure to capture semantic properties of rare words compared to frequent ones.

Methodology

To address this, the researchers bridge the gap between static word embeddings and deep contextualized models using two primary techniques:

One-Token Approximation (OTA)

OTA is the core technical contribution of this work. Since BERT lacks single embeddings for words split into subwords, OTA infers what such an embedding would look like. It searches for a single vector that, when inserted into a sentence, exerts the same influence on the surrounding context as the original sequence of subword tokens.

Attentive Mimicking (AM)

Using these "synthetic" single-token embeddings, a mimicking model is trained to predict a word's embedding based on its surface form (characters/n-grams) and its contexts (sentences where it appears). AM utilizes a self-attention mechanism to distinguish between highly informative contexts and unhelpful noise. By training AM to "mimic" the embeddings of frequent words, it can generate high-quality embeddings for rare words the model has rarely encountered.

Evaluation and Datasets

The study introduces WNLaMPro (WordNet Language Model Probing), a novel dataset designed to evaluate word understanding without task-specific fine-tuning. It uses WordNet to create "cloze" tasks (fill-in-the-blank) across four relations: Antonyms, Hypernyms, Cohyponyms, and Corruptions (typos). Additionally, the Westbury Wikipedia Corpus (WWC) provides the necessary contexts for training the AM model.

Key Results

The evaluation on WNLaMPro reveals a stark correlation between word frequency and model understanding. Performance drops significantly for words appearing fewer than 256 times in the training data.

Mean Reciprocal Rank (MRR) across different word frequency subsets in WNLaMPro, comparing BERT_base and the proposed OTA method.

Adding AM to BERT leads to massive gains:

  • Antonyms: BERT-base's Mean Reciprocal Rank (MRR) improved from 0.149 to 0.449.
  • Handling Typos: In the "Corruption" category, MRR jumped from 0.020 to 0.254, demonstrating the power of surface-form mimicking.
  • Consistency: Improvements were consistent across BERT-base and BERT-large for rare and medium-frequency words.

Conclusion

This work proves that mimicking, previously used for static embeddings, can be successfully adapted to complex Transformer architectures. By leveraging OTA and AM, models can achieve a more robust understanding of specialized terms and misspelled words in a zero-shot manner. Future research may explore more complex AM architectures and the impact of these improvements on downstream tasks like Question Answering.

Summary updated

Abstract

Pretraining deep neural network architectures with a language modeling objective has brought large improvements for many natural language processing tasks. Exemplified by BERT, a recently proposed such architecture, we demonstrate that despite being trained on huge amounts of data, deep language models still struggle to understand rare words. To fix this problem, we adapt Attentive Mimicking, a method that was designed to explicitly learn embeddings for rare words, to deep language models. In order to make this possible, we introduce one-token approximation, a procedure that enables us to use Attentive Mimicking even when the underlying language model uses subword-based tokenization, i.e., it does not assign embeddings to all words. To evaluate our method, we create a novel dataset that tests the ability of language models to capture semantic properties of words without any task-specific fine-tuning. Using this dataset, we show that adding our adapted version of Attentive Mimicking to BERT does substantially improve its understanding of rare words.
BibTeX
@inproceedings{schick2020rare,
  author = {Timo Schick and Hinrich Schütze},
  title = {{Rare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive Mimicking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=bhrSjr-nPO7},
}
Figure