Abstract
Estimator Vectors (EV) is a novel neural network architecture designed to address the "Out-of-Vocabulary" (OOV) problem by jointly learning word, subword, and "context clue" embeddings. By training these representations simultaneously, the model produces high-quality word vectors and can accurately estimate the meaning of unknown words using both their internal morphological structure and surrounding context. Our evaluation shows that EV outperforms standard models in semantic tasks and provides superior OOV estimation as context availability increases.
Introduction
Classic Natural Language Processing (NLP) models, such as word2vec or GloVe, are constrained by a fixed vocabulary. When encountering an OOV word—such as a technical term, neologism, or misspelling—these models typically default to a generic "unknown" token, losing critical semantic nuance. In contrast, humans infer the meaning of new words by analyzing sub-components (e.g., prefixes) and the surrounding sentence. While previous methods have utilized subwords (like fastText) or context (like à la carte embeddings), they often treat these as disjoint post-processing steps. This paper proposes Estimator Vectors (EV), a unified model that learns these relationships from the ground up.
Methodology
The EV architecture departs from the standard skip-gram model by maintaining three distinct sets of vectors:
- Word Embeddings ($v$): Standard semantic representations.
- Context Clue Embeddings ($h$): Specialized vectors used to estimate a target word based on its neighbors.
- Subword Embeddings ($z$): Vectors for character n-grams used to estimate a word based on its spelling.
Joint Training
Unlike previous approaches that rely on pre-trained vectors, EV trains all three sets jointly. During training, the model predicts a context word using two distinct estimates of the target word: an average of the context clue embeddings of its neighbors and an average of its subword embeddings. This interplay ensures that word embeddings are enriched by both subword and context information, while the estimator vectors are specifically optimized to produce accurate word-level representations.
Post-processing
To refine the results, a "Context Clue" post-processing step is applied, removing the top three Principal Component Analysis (PCA) components from the context clue sums. This filters out common "background noise" in the vector space, highlighting the unique semantic signal of the OOV word.
Experimental Evaluation
The model was trained on a modified version of the Westbury Wikipedia Corpus (WWC) and evaluated across several benchmarks, including word analogy tasks (semantic and syntactic), the WS353 similarity dataset, and the Contextualized Rare Word (CRW) task, which evaluates OOV estimation across varying amounts of context.
Results and Analysis
EV-word vectors demonstrated superior semantic quality compared to skip-gram and fastText. As shown in the comparison below, EV achieved 85.30% on semantic analogies, significantly outperforming the baselines.
While EV excelled in semantics, it showed a slight trade-off in syntactic performance compared to fastText. This is likely because subword-only models are naturally biased toward structural and morphological patterns.
In the OOV estimation domain, the CRW task revealed that EV becomes increasingly effective as more context is provided. Once at least four context words are available, EV outperforms competing methods, including the state-of-the-art Form-Context model.
Conclusion and Future Work
Estimator Vectors provide a robust solution to the OOV problem by integrating subword and context information through joint learning. The separation of a word's meaning from its utility as a context clue allows for more informative neighbor representations. Future work will explore gated weighting strategies and comparisons with large-scale transformer models to further refine lightweight OOV estimation.
Summary updated