In the early 2010s, Natural Language Processing (NLP) underwent a revolution driven by the distributional hypothesis: the idea that a word’s meaning can be inferred from the company it keeps. This led to the era of static word embeddings, where models like Glove: Global Vectors for Word Representation by Jeffrey Pennington, Christopher D Manning, and Richard Socher mapped every word in a vocabulary to a single, fixed-dimensional vector.
While powerful, these static models hit a fundamental ceiling: polysemy. In a static world, the word "bank" has one vector, whether it refers to a financial institution or the edge of a river. This "one-vector-per-word" bottleneck forces the model to conflate disparate meanings into a single point in space, blurring the semantic signal.
The Static Bottleneck and the Polysemy Problem
Static embeddings are essentially a lookup table. When a model sees a token, it retrieves a pre-calculated vector regardless of the surrounding text. As noted in Word Representation by Zhiyuan Liu and colleagues, this invariant nature prevents models from capturing shifts in meaning.
The consequences are visible across diverse domains:
- Medical Data: In cui2vec, Andrew L. Beam and others found that static points fail to disambiguate complex clinical scenarios where a single term might have multiple implications.
- Machine Translation: Handling Homographs in Neural Machine Translation demonstrates that translation quality drops as the number of senses for a word increases, because the model is forced to disambiguate meanings only after the initial semantic information has been "blurred" by the static lookup.
- Information Retrieval: Sajad Movahedi and Azadeh Shakery show in QEQR that static expansion suggests irrelevant synonyms (like "electrical charge" for a "legal charge") because it lacks context-sensitivity.
From Points to Functions: The Contextual Shift
The solution was to move from a lookup table to a function. Instead of a word having a fixed vector, its representation became a dynamic output of a neural network that considers the entire sentence.
Early hints of this appeared in A Simple Word Embedding Model for Lexical Substitution, where Omer Levy and Ido Dagan realized that the "context vectors" discarded during Word2Vec training actually held the key to finding context-appropriate synonyms.
The true breakthrough came with models like ELMo and BERT. By using deep architectures (LSTMs or Transformers), these models generate "contextualized" embeddings. As Wanxiang Che and Ting Liu demonstrated in the CoNLL 2018 Shared Task, using deep contextualized embeddings allows parsers to resolve syntactic ambiguities that static vectors simply cannot see. Similarly, Sarik Ghazarian showed that BERTScore correlates better with human judgment because it captures the nuanced semantics of a specific conversational turn rather than just matching keywords.
Why Contextualization Works: Dynamic Weighting
Intuition for why this works can be found in the mechanism of attention. In a contextualized model, the representation of a word is a weighted sum of its neighbors.
In SPARC, Jinhyuk Lee and Minjoon Seo use rectified self-attention to dynamically weight n-grams. If the word "bank" appears near "river" and "water," the attention mechanism pulls the representation toward the "geographic" region of the semantic space. If it appears near "interest" and "account," it shifts toward the "financial" region.
This allows the model to represent uncertainty and range. While some researchers like Chi Sun and Xipeng Qiu explored modeling words as Gaussian distributions to capture this uncertainty, the transformer's ability to simply "move" the vector to the correct spot in high-dimensional space proved more scalable.
Recovering the Lexical Core
If contextualized models are so much better, why do we still talk about static embeddings? Yu Meng and Jiaxin Huang argue in Spherical Text Embedding that static embeddings remain necessary for tasks requiring a stable semantic identity, such as word similarity benchmarks where there is no context to provide.
Recent research has focused on "distilling" the best of both worlds. Ivan Vulić and Edoardo Ponti have shown that we can "probe" pretrained language models to extract high-quality static vectors. In Probing Pretrained Language Models for Lexical Semantics, they found that averaging representations from the lower layers of BERT recovers "decontextualized" word vectors that outperform traditional methods like fastText. Their follow-up work, LexFit, introduces a fine-tuning procedure to "rewire" these models specifically for static lexical tasks.
Beyond General Context: Covariates and Domains
The final frontier of this direction is capturing context that isn't just linguistic, but situational. Kevin Tian and James Zou introduced CoVeR, which learns how word meanings shift across covariates like time or political leaning. This recognizes that "context" isn't just the words next to a token, but the person saying it and the era in which it was said.
By moving from static points to dynamic, context-aware functions, the field has transitioned from simply cataloging words to understanding the fluid, living nature of human language.