Learning Semantic Representations for Novel Words: Leveraging Both Form and Context.

Timo Schick, Hinrich Schütze

Abstract

Word embeddings are a key component of high-performing natural language processing (NLP) systems, but it remains a challenge to learn good representations for novel words on the fly, i.e., for words that did not occur in the training data. The general problem setting is that word embeddings are induced on an unlabeled training corpus and then a model is trained that embeds novel words into this induced embedding space. Currently, two approaches for learning embeddings of novel words exist: (i) learning an embedding from the novel word’s surface-form (e.g., subword n-grams) and (ii) learning an embedding from the context in which it occurs. In this paper, we propose an architecture that leverages both sources of information – surface-form and context – and show that it results in large increases in embedding quality. Our architecture obtains state-of-the-art results on the Definitional Nonce and Contextual Rare Words datasets. As input, we only require an embedding set and an unlabeled corpus for training our architecture to produce embeddings appropriate for the induced embedding space. Thus, our model can easily be integrated into any existing NLP system and enhance its capability to handle novel words.

Introduction

Standard Natural Language Processing (NLP) systems rely on word embeddings—dense vectors that represent word meanings. However, these systems face a "cold start" problem: they struggle with novel, out-of-vocabulary (OOV) words that did not appear in their training data.

Existing solutions usually fall into two categories. Surface-form approaches analyze the characters or sub-components (morphemes/n-grams) of a word; for instance, one can infer the meaning of "unemployable" from its constituent parts. Context-based approaches look at the words surrounding the novel word. If a sentence reads, "A cardigan is a knitted jacket...", the meaning of "cardigan" becomes clear even if the word itself is unfamiliar. This paper argues that relying on only one source is insufficient. Some words have clear structures but vague contexts, while others have opaque structures but clear contexts. The challenge lies in building a model that can leverage both sources and dynamically determine which to trust.

The Form-Context Model

The proposed Form-Context Model utilizes a gated neural architecture to intelligently weight information from a word's internal structure and its surrounding context. The process follows three primary steps:

  1. Surface-Form Embedding: The model decomposes a word into character n-grams and averages their learned embeddings.
  2. Context Embedding: The model averages the embeddings of words surrounding the novel word and applies a learned linear transformation ($A$) to align them with the target embedding space.
  3. Gated Combination: Instead of simple averaging, the model employs a gate ($\alpha$). This gate is a function of both the form and context vectors, allowing the model to dynamically decide which source is more reliable for a specific word.
The Form-Context Model architecture showing the integration of subword n-grams and context word embeddings via a gating mechanism.

The model is trained to reconstruct existing, high-quality embeddings from a large unlabeled corpus (the Westbury Wikipedia Corpus), making it easily integrable into pre-existing NLP pipelines.

Experimental Evaluation

The researchers evaluated the model on two primary benchmarks:

  • Definitional Nonce (DN): Consists of 300 test words, each provided with a single definitional sentence from Wikipedia.
  • Contextual Rare Words (CRW): Features 562 pairs of rare words with human similarity judgments and 255 context sentences per word.

Key Results

The hybrid model significantly outperformed models using only form or only context. On the Definitional Nonce dataset, the gated model achieved a Median Rank of 49. This implies that for half of the test words, the model's predicted embedding was among the top 0.02% most similar words in a vocabulary of approximately 260,000. This represents a substantial improvement over the previous state-of-the-art (A La Carte), which had a median rank of 165.5. Furthermore, the gated model reached a Mean Reciprocal Rank (MRR) of 0.175, nearly doubling the performance of purely context-based (0.065) or purely form-based (0.129) baselines.

Performance comparison on the CRW dataset showing Spearman's rho correlation as a function of the number of contexts. The form-context (frm-ctx) model consistently outperforms individual baselines.

On the CRW dataset, the model demonstrated remarkable efficiency. As shown in the results, the form-context model achieved performance comparable to standard skip-gram models (trained on 255 contexts) using as few as 8 contexts. This suggests that even when a word is morphologically complex, adding even a small amount of context helps refine the representation.

Conclusion

The success of the Form-Context Model highlights the complementary nature of morphological and contextual information. The gated architecture allows the model to learn that certain subword patterns (like specific suffixes) are highly reliable, while more ambiguous terms require heavier weighting of context.

Future work may involve refining the context processing, such as weighting sentences based on their "definitional quality" or utilizing more sophisticated composition functions like Transformers to replace simple averaging. Additionally, integrating positional information could further enhance the model's ability to capture nuanced semantic meanings.

Summary updated

Abstract

Word embeddings are a key component of high-performing natural language processing (NLP) systems, but it remains a challenge to learn good representations for novel words on the fly, i.e., for words that did not occur in the training data. The general problem setting is that word embeddings are induced on an unlabeled training corpus and then a model is trained that embeds novel words into this induced embedding space. Currently, two approaches for learning embeddings of novel words exist: (i) learning an embedding from the novel word’s surface-form (e.g., subword n-grams) and (ii) learning an embedding from the context in which it occurs. In this paper, we propose an architecture that leverages both sources of information – surface-form and context – and show that it results in large increases in embedding quality. Our architecture obtains state-of-the-art results on the Definitional Nonce and Contextual Rare Words datasets. As input, we only require an embedding set and an unlabeled corpus for training our architecture to produce embeddings appropriate for the induced embedding space. Thus, our model can easily be integrated into any existing NLP system and enhance its capability to handle novel words.
BibTeX
@inproceedings{schick2019learning,
  author = {Timo Schick and Hinrich Schütze},
  title = {{Learning Semantic Representations for Novel Words: Leveraging Both Form and Context.}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=rm5slMx_TH},
}
Figure