Introduction
This research challenges a foundational assumption in Natural Language Processing (NLP): that sense-specific word embeddings—which assign different vectors to polysemous words based on context (e.g., "bank" as a river edge vs. a financial institution)—improve performance by capturing linguistic nuance. Instead, the authors demonstrate that these performance gains are largely a statistical artifact of sub-sampling, which reduces bias in cosine distance estimation. Remarkably, random sense assignments often perform as well as, or better than, linguistically grounded ones.
The Problem with Polysemy Evaluation
Standard NLP models like word2vec represent each word with a single dense vector. However, because many words are polysemous, researchers have increasingly turned to "sense-aware" models. These models were thought to be superior because they achieved higher scores on "gold standard" word similarity tasks, where models rank word pairs based on human-judged similarity. The authors investigated whether these gains truly stem from a better understanding of polysemy or if word similarity tasks are simply flawed proxies for evaluating multiple word senses.
Methodology: Controls and Pseudo-words
The researchers used a multi-pronged approach to isolate the effects of polysemy:
- Polysemy Induction: They created "pseudo-words" by merging unrelated tokens (e.g., "ring" and "table") to simulate perfect, known polysemy.
- Sense-Annotated Training: Using the OntoNotes corpus, they trained models with ground-truth sense labels.
- Random Control Condition: They assigned word occurrences to "senses" completely at random. If performance still improved, the gain could not be attributed to linguistic structure.
- Theoretical Analysis: They analyzed the bias of the cosine distance estimator when data is split into smaller subsets (senses).
Key Insights: The Sub-sampling Artifact
A critical finding is that true polysemy often acts as a penalty rather than a bonus in similarity tasks. When testing models on controlled datasets, the "actual gain" from capturing polysemy was frequently negative or negligible compared to the "reported gains" in literature.
The "aha moment" lies in the sub-sampling artifact. Sense-specific models estimate a word's vector using smaller subsets of data. Averaging these multiple estimations reduces the statistical bias inherent in the cosine distance formula. Consequently, the more a model's "senses" resemble random noise, the better it often performs on evaluation benchmarks—a phenomenon the authors call the "Polysemic Signature."
Results: The Power of Randomness
The empirical results were striking. On the WordSim-353 task using OntoNotes, a single-vector "Global" model scored 44.7, while the sense-specific "Average" model dropped to 41.3. More importantly, the "Random Boost" was replicated: assigning senses randomly to a regular embedding increased its score on the SCWS task from 65.7 to 67.3, matching the performance of complex, state-of-the-art models.
As shown in the table above, random sense assignment consistently yields performance comparable to or higher than the original sense-aware implementations, confirming that the reported benefits are often mathematical quirks rather than linguistic breakthroughs.
Conclusion
While sense-specific vectors may still hold value for downstream tasks like Machine Translation, this research serves as a cautionary tale. Reported improvements on similarity tasks must be validated against random-sense controls. The NLP community requires more principled evaluation sets designed specifically for polysemy, moving beyond proxies that are easily manipulated by statistical noise.
Summary updated