Abstract
Pretraining deep neural network architectures with a language modeling objective has brought large improvements for many natural language processing tasks. Exemplified by BERT, a recently proposed such architecture, we demonstrate that despite being trained on huge amounts of data, deep language models still struggle to understand rare words. To fix this problem, we adapt Attentive Mimicking, a method that was designed to explicitly learn embeddings for rare words, to deep language models. In order to make this possible, we introduce one-token approximation, a procedure that enables us to use Attentive Mimicking even when the underlying language model uses subword-based tokenization, i.e., it does not assign embeddings to all words. To evaluate our method, we create a novel dataset that tests the ability of language models to capture semantic properties of words without any task-specific fine-tuning. Using this dataset, we show that adding our adapted version of Attentive Mimicking to BERT does indeed substantially improve its understanding of rare words.
Introduction: The Achilles' Heel of BERT
Modern Natural Language Processing (NLP) relies heavily on contextualized embeddings from models like BERT. While these models appear to possess encyclopedic knowledge, their understanding is fundamentally tied to word frequency. Rare words—those appearing infrequently in training corpora—present a significant challenge for even the most advanced architectures.
This "rare word" problem stems from two main issues:
- Subword Tokenization: To handle an open vocabulary, models use subword tokenization (e.g., WordPiece). Rare words are often fractured into meaningless fragments (e.g., "unicycle" becomes
un,ic,y,cle), making it difficult for the model to synthesize a coherent semantic concept. - Data Scarcity: Even when not split, words with few occurrences provide insufficient signal for the model to learn nuanced meanings during standard pretraining.
Methodology: Attentive Mimicking and One-Token Approximation
To address these limitations, the authors adapt Attentive Mimicking (AM). AM generates high-quality embeddings for rare words by attending to both their surface form (character n-grams) and their context (surrounding sentences).
One-Token Approximation (OTA)
A primary challenge in applying AM to BERT is that AM requires a single "gold" target embedding to mimic. Since BERT uses subwords, many words lack a single representative vector. The authors introduce One-Token Approximation (OTA) to bridge this gap. For any multi-token word, OTA identifies a single synthetic vector that, when substituted for the subword sequence, minimizes the difference in the model's internal hidden states. This synthetic vector serves as the ground truth for the AM model.
AM+CONTEXT
The method is further refined by incorporating n-gram dropout and increasing training samples. This forces the model to rely more heavily on contextual clues rather than just orthographic patterns, leading to more robust semantic representations.
Probing Semantic Knowledge: The WNLaMPro Dataset
To evaluate these methods, the paper introduces WNLaMPro (WordNet Language Model Probing). This dataset consists of "cloze" tasks derived from WordNet, testing four key semantic relations: Antonyms, Hypernyms, Cohyponyms, and Corruptions (misspellings). For example, a model might be asked to complete: "A lime is a ____."
Key Insights: Frequency as a Performance Ceiling
Analysis reveals that BERT's performance is strictly bounded by word frequency. If a word appears fewer than 100–200 times in the training corpus, the model's ability to identify its basic properties drops significantly.
Results
The integration of Attentive Mimicking with BERT yields substantial improvements, particularly for rare words and misspellings.
- Antonyms: On the "Rare" subset of WNLaMPro, BERT's Mean Reciprocal Rank (MRR) improved from 0.149 to 0.449 with AM.
- Corruptions: BERT is highly fragile to typos (e.g., "resigntaion"), achieving an MRR of only 0.020. BERT+AM increased this to 0.254, demonstrating the robustness of character-level mimicking.
Conclusion
Attentive Mimicking, powered by One-Token Approximation, serves as an effective "patch" for deep language models. It allows for the generation of high-quality embeddings for rare words without requiring expensive retraining of the entire model. While currently demonstrated on English, this approach offers a promising path toward more robust language models that can handle the long tail of human vocabulary.
Summary updated