Learning Semantic Representations for Novel Words: Leveraging Both Form and Context

Timo Schick, Hinrich Schütze

Abstract Word embeddings are central to modern NLP, yet representing "novel" words—those absent from training data—remains a significant challenge. Existing methods typically rely either on a word's surface-form (e.g., subword n-grams) or the context in which it appears. This paper introduces the Form-Context Model (FCM), an architecture that intelligently combines both information sources. By leveraging a gated neural network, the FCM achieves state-of-the-art results on the Definitional Nonce and Contextual Rare Words benchmarks, providing a robust solution for handling out-of-vocabulary (OOV) terms in any embedding space.

The Challenge of Out-of-Vocabulary Words

Standard NLP systems often operate with a fixed vocabulary. When encountering an OOV word, they typically default to a generic "unknown" token, losing critical semantic information. Current strategies to mitigate this follow two distinct paths:

  1. Surface-form approaches: These infer meaning from spelling and morphology (e.g., identifying "un-employ-able" as a derivative of "employ").
  2. Context-based approaches: These infer meaning from surrounding words (e.g., deducing that a "pomelo" is a fruit because it appears near "citrus" and "peel").

However, both methods have limitations. Spelling can be deceptive (e.g., "spies" contains "pies"), and context can be ambiguous or sparse. The FCM addresses this by determining when to trust the form and when to trust the context.

The Form-Context Model (FCM)

The FCM architecture consists of two parallel branches whose outputs are merged through a learned gating mechanism.

The Form-Context Model (FCM) architecture, showing the integration of subword n-grams (form) and context embeddings via a gated mechanism.
  • Form Branch: This branch decomposes a word into character n-grams (lengths 3–5). The embeddings of these n-grams are averaged to produce a surface-form representation, allowing the model to generalize across morphological variants and handle minor typos.
  • Context Branch: This branch averages the embeddings of words in the target word's immediate neighborhood and applies a learned linear transformation to capture thematic relevance.
  • Gated Mechanism: The core innovation is a gate ($\alpha$), a small neural network that evaluates both the form and context vectors. It outputs a weight between 0 and 1, dynamically balancing the contribution of each branch based on the specific word and context provided.

The model is trained using a "mimicry" objective: it learns to reconstruct the embeddings of known words from a large corpus (like Wikipedia) using only their spelling and a sample of their contexts.

Experimental Evaluation

The researchers evaluated the FCM on two key benchmarks:

  • Definitional Nonce (DN): Testing the ability to recreate a word's embedding from a single Wikipedia definition.
  • Contextual Rare Words (CRW): Measuring the correlation between model-generated embeddings and human similarity judgments for rare words.

The model was trained on the Westbury Wikipedia Corpus (WWC). To improve evaluation, the authors also introduced a new 550-pair development set for the CRW task.

Key Results and Insights

The FCM significantly outperformed existing baselines. On the Definitional Nonce task, the gated model achieved a Median Rank of 49, drastically improving upon previous state-of-the-art models like A La Carte (165.5) and Nonce2Vec (623).

Performance comparison on the Definitional Nonce dataset. The gated FCM model achieves the lowest Median Rank and highest MRR.

Complementary Strengths: Qualitative analysis revealed that the gate successfully mitigates the weaknesses of individual branches. For a word like "spies," the form branch might be misled by the subword "pies," but the context branch (seeing "espionage" or "clandestine") corrects the representation. Conversely, for "hygiene," the context might be a vague mythological reference, but the surface form provides a clear signal.

Data Efficiency: The FCM is remarkably efficient. On the CRW dataset, it achieved higher accuracy with only 8 contexts than standard Skipgram models achieved with 255 contexts. This makes it an ideal candidate for "few-shot" learning of new vocabulary.

Conclusion

The Form-Context Model demonstrates that "looking" at a word and "listening" to its neighbors are complementary strategies. By dynamically weighting these inputs, the FCM provides a flexible and powerful tool for extending existing embedding spaces to handle novel words. Future work may involve more sophisticated architectures like Transformers to capture complex contextual relationships or incorporating positional information to further refine semantic accuracy.

Summary updated

Abstract

Word embeddings are a key component of high-performing natural language processing (NLP) systems, but it remains a challenge to learn good representations for novel words on the fly, i.e., for words that did not occur in the training data. The general problem setting is that word embeddings are induced on an unlabeled training corpus and then a model is trained that embeds novel words into this induced embedding space. Currently, two approaches for learning embeddings of novel words exist: (i) learning an embedding from the novel word's surface-form (e.g., subword n-grams) and (ii) learning an embedding from the context in which it occurs. In this paper, we propose an architecture that leverages both sources of information - surface-form and context - and show that it results in large increases in embedding quality. Our architecture obtains state-of-the-art results on the Definitional Nonce and Contextual Rare Words datasets. As input, we only require an embedding set and an unlabeled corpus for training our architecture to produce embeddings appropriate for the induced embedding space. Thus, our model can easily be integrated into any existing NLP system and enhance its capability to handle novel words.
BibTeX
@inproceedings{schick2019learning,
  author = {Timo Schick and Hinrich Schütze},
  title = {{Learning Semantic Representations for Novel Words: Leveraging Both Form and Context}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=rm5slMx_TH},
}
Figure