Abstract Word embeddings are central to modern NLP, yet representing "novel" words—those absent from training data—remains a significant challenge. Existing methods typically rely either on a word's surface-form (e.g., subword n-grams) or the context in which it appears. This paper introduces the Form-Context Model (FCM), an architecture that intelligently combines both information sources. By leveraging a gated neural network, the FCM achieves state-of-the-art results on the Definitional Nonce and Contextual Rare Words benchmarks, providing a robust solution for handling out-of-vocabulary (OOV) terms in any embedding space.
The Challenge of Out-of-Vocabulary Words
Standard NLP systems often operate with a fixed vocabulary. When encountering an OOV word, they typically default to a generic "unknown" token, losing critical semantic information. Current strategies to mitigate this follow two distinct paths:
- Surface-form approaches: These infer meaning from spelling and morphology (e.g., identifying "un-employ-able" as a derivative of "employ").
- Context-based approaches: These infer meaning from surrounding words (e.g., deducing that a "pomelo" is a fruit because it appears near "citrus" and "peel").
However, both methods have limitations. Spelling can be deceptive (e.g., "spies" contains "pies"), and context can be ambiguous or sparse. The FCM addresses this by determining when to trust the form and when to trust the context.
The Form-Context Model (FCM)
The FCM architecture consists of two parallel branches whose outputs are merged through a learned gating mechanism.
- Form Branch: This branch decomposes a word into character n-grams (lengths 3–5). The embeddings of these n-grams are averaged to produce a surface-form representation, allowing the model to generalize across morphological variants and handle minor typos.
- Context Branch: This branch averages the embeddings of words in the target word's immediate neighborhood and applies a learned linear transformation to capture thematic relevance.
- Gated Mechanism: The core innovation is a gate ($\alpha$), a small neural network that evaluates both the form and context vectors. It outputs a weight between 0 and 1, dynamically balancing the contribution of each branch based on the specific word and context provided.
The model is trained using a "mimicry" objective: it learns to reconstruct the embeddings of known words from a large corpus (like Wikipedia) using only their spelling and a sample of their contexts.
Experimental Evaluation
The researchers evaluated the FCM on two key benchmarks:
- Definitional Nonce (DN): Testing the ability to recreate a word's embedding from a single Wikipedia definition.
- Contextual Rare Words (CRW): Measuring the correlation between model-generated embeddings and human similarity judgments for rare words.
The model was trained on the Westbury Wikipedia Corpus (WWC). To improve evaluation, the authors also introduced a new 550-pair development set for the CRW task.
Key Results and Insights
The FCM significantly outperformed existing baselines. On the Definitional Nonce task, the gated model achieved a Median Rank of 49, drastically improving upon previous state-of-the-art models like A La Carte (165.5) and Nonce2Vec (623).
Complementary Strengths: Qualitative analysis revealed that the gate successfully mitigates the weaknesses of individual branches. For a word like "spies," the form branch might be misled by the subword "pies," but the context branch (seeing "espionage" or "clandestine") corrects the representation. Conversely, for "hygiene," the context might be a vague mythological reference, but the surface form provides a clear signal.
Data Efficiency: The FCM is remarkably efficient. On the CRW dataset, it achieved higher accuracy with only 8 contexts than standard Skipgram models achieved with 255 contexts. This makes it an ideal candidate for "few-shot" learning of new vocabulary.
Conclusion
The Form-Context Model demonstrates that "looking" at a word and "listening" to its neighbors are complementary strategies. By dynamically weighting these inputs, the FCM provides a flexible and powerful tool for extending existing embedding spaces to handle novel words. Future work may involve more sophisticated architectures like Transformers to capture complex contextual relationships or incorporating positional information to further refine semantic accuracy.
Summary updated