Introduction
Standard distributional word embeddings, such as word2vec and GloVe, have become fundamental tools in natural language processing. However, these models suffer from a significant "data hunger" problem: they typically require hundreds of occurrences of a word to produce a reliable semantic vector. For rare or previously unseen words, the resulting embeddings are often noisy and lack utility.
Furthermore, researchers often require embeddings for specific linguistic features, such as bigrams (e.g., "New York") or WordNet synsets. Traditionally, obtaining these requires a "prix fixe" computational investment: the entire model must be retrained on a massive corpus where these features are treated as unique tokens. Simple "on-the-fly" alternatives, like averaging the vectors of surrounding words (the additive approach), often fail because frequent "stop words" dominate the signal, obscuring the specific meaning of the target feature.
Methodology: The A La Carte Approach
The authors propose a "shallow" extension of the distributional hypothesis. Rather than assuming a feature's embedding is the average of its context, they posit that a feature's true semantic representation is a linear transformation of that average.
The method involves three primary steps:
- The Transform: A matrix $A$ is defined such that for a word $w$, its pretrained vector $v_w \approx A \cdot u_w$, where $u_w$ is the average of the vectors of words appearing in $w$'s context.
- Learning $A$: Since high-quality pretrained vectors already exist for frequent words, learning $A$ is treated as a large-scale linear regression problem. The known vectors of frequent words serve as targets, and their context averages serve as inputs.
- Induction: Once $A$ is learned and fixed, it can induce an embedding for any new feature $f$ (e.g., a rare word or bigram) by simply multiplying the average of the vectors in its available contexts by $A$.
This "a la carte" approach allows for the targeted computation of specific embeddings as needed for downstream tasks, bypassing the need for full-model retraining.
Evaluation and Datasets
To test the versatility of the method, the authors utilized several benchmarks, including the newly introduced Contextual Rare Words (CRW) dataset. CRW consists of 562 pairs from the Rare Word (RW) dataset, each supplemented with 255 context sentences from Wikipedia. This allows for a granular analysis of how embedding quality improves as more context becomes available.
As shown in the evaluation on the CRW task, the à la carte method consistently outperforms additive baselines and specialized weighting schemes (like SIF), particularly as the number of available contexts increases.
Key Results
The A La Carte method demonstrates state-of-the-art performance across several "few-shot" learning tasks, where models must induce meaning from very limited data.
On the Nonce task, which requires learning a word's meaning from a single definition, A La Carte achieved a Median Rank of 165.5. This represents a massive improvement over the previous state-of-the-art, nonce2vec, which had a Median Rank of 623. Similarly, the method proved effective for Word Sense Disambiguation, outperforming the "Most Frequent Sense" baseline on SemEval tasks by learning representations from the relatively small SemCor dataset.
Beyond rare words, the method excels in Document Classification. By concatenating unigram, bigram, and trigram embeddings induced via A La Carte, the authors achieved results competitive with complex, deep unsupervised models. For instance, on the IMDB sentiment task, they reached 90.9% accuracy, rivaling models like skip-thoughts that are significantly more expensive to train.
Key Insights and Discussion
The effectiveness of the learned matrix $A$ stems from its role as a data-driven de-noiser. It effectively learns to ignore common directions in the vector space associated with stop words and grammatical filler, acting as an automated version of Smooth Inverse Frequency (SIF) weighting.
The success of a single global linear transformation reinforces recent theoretical findings suggesting that the semantic space of word embeddings possesses a surprisingly linear structure. Furthermore, by focusing on context over composition, A La Carte successfully captures the meaning of idiomatic expressions (e.g., "kick the bucket") that purely compositional methods fail to represent accurately.
Limitations and Future Work
While powerful, the current formulation relies purely on context and does not incorporate morphological information (e.g., sub-word structures like prefixes or suffixes). Integrating A La Carte with sub-word models like FastText could further enhance results for rare words. Future research may also explore more complex context structures, such as dependency parses, or "zero-shot" learning scenarios where no context is available.
Summary updated