In early natural language processing models, words were treated as static points in a high-dimensional space. Algorithms like Word2Vec or GloVe assigned a single, fixed vector to each vocabulary word. While powerful, this approach suffered from a fundamental flaw: it assumed a word's meaning was invariant. In reality, language is highly fluid. The word "bank" means something entirely different in "investment bank" than it does in "river bank."
To resolve this, the field shifted toward contextualized word representations. Instead of looking up a word in a static dictionary, modern models generate word vectors on the fly as a function of the entire sentence. This paradigm allows a single word form to take on infinitely many vector representations depending on its surrounding linguistic environment.
The Tyranny of the Static Vector
To understand why contextualized representations work, we must first understand why static embeddings fail. A static embedding forces a model to compress all possible meanings, grammatical functions, and pragmatic nuances of a word into a single coordinate.
When a word has multiple meanings (polysemy), its static vector ends up being a geometric compromise—an average of all its uses that fits none of them perfectly. If a model encounters a rare or figurative use of a word, the static embedding pulls the representation back toward the dominant, frequent meaning.
Early attempts to bypass this limitation modified static vectors using local context. For instance, Apidianaki demonstrated that static embeddings could be dynamically adjusted by modifying a target word's vector based on the embeddings of its immediate neighbors. Similarly, Kiros et al. introduced multiplicative gating units, where external metadata or domain attributes explicitly re-wired word embeddings to reflect shifting semantic domains. While these methods helped, they were constrained by the underlying rigidity of the static lookup table.
From Lookups to Sequence Functions
The true breakthrough came from treating word representation not as a retrieval task, but as a sequence processing task. Instead of using a static lookup table, an auxiliary neural network processes the entire sentence to compute representations dynamically.
McCann et al. pioneered this with Contextualized Word Vectors (CoVe), utilizing a deep machine translation encoder to generate context-sensitive vectors. By training on a sequence-to-sequence task, the model learned to output vectors that varied based on surrounding words, successfully resolving lexical ambiguities. As Bisazza observed, this architectural shift means that crucial linguistic features—such as morphology, tense, and gender—are generated through contextual interaction within the hidden states of the network, rather than being retrieved from a fixed lexical repository.
Static Lookup:
"bank" ──> [0.25, -0.41, 0.88] (Same vector for finance and rivers)
Dynamic Sequence Function:
"The river bank..." ──> [Encoder] ──> "bank" = [0.11, 0.72, -0.34]
"The investment bank..." ──> [Encoder] ──> "bank" = [0.89, -0.12, 0.55]
Subsequent architectures like ELMo and BERT scaled this concept by using deep bidirectional language modeling objectives. As shown by Chen et al., combining global dictionary meanings with these local, dynamically generated contextual features allows models to accurately distinguish between different senses of the same word on the fly.
How the Vector Space Behaves
What does a contextualized vector space actually look like? Instead of a single point per word, each token inhabits a dynamic "cloud" or manifold of points.
An et al. measured this behavior by calculating the dot product between target tokens and predefined semantic vectors across different sentence environments. They found that surrounding information dynamically shifts the internal representation of a token toward specific semantic poles. For example, placing a gender-neutral name next to an occupational stereotype literally pulls the name's vector closer to a gendered region of the embedding space.
Crucially, this movement is not random; it balances flexibility with stability. Bianchi et al. demonstrated that contextual representations maintain a stable "semantic core" for words while modulating them toward specific synonyms based on the sentence. To formalize this mathematically, Nagata reformulated Zipf's meaning-frequency law by generating unique embeddings for every single instance of a token, using the spatial diversity of these vectors to measure polysemy directly from raw text.
Resolving Linguistic Ambiguity
By allowing vectors to shift based on context, models can effortlessly resolve linguistic phenomena that previously paralyzed NLP systems.
Grammatical and Structural Ambiguity
Identical word forms often play entirely different grammatical roles. El-Kahky et al. showed that contextualized representations easily handle noun-verb ambiguities (such as distinguishing the noun "flies" from the verb "flies"). Because the model processes the entire sentence layout, the local syntactic environment reshapes the token's vector to reflect its correct part of speech before it reaches downstream classifiers.
Figurative and Pragmatic Shifts
In figurative language, the true meaning of a word diverges sharply from its literal definition. Choi et al. demonstrated that contextualized embeddings are highly effective for metaphor detection because they capture semantic variations where a word's meaning shifts based on its neighbors.
This applies equally to sarcasm, irony, and pragmatics. Ilic et al. found that dynamic embeddings are essential for sarcasm detection because sarcasm relies on a semantic clash between adjacent words; contextualized models alter the word representations to reflect this tension. Similarly, Aljanaideh et al. used contextualized clusters to map how the word "please" shifts from a polite request to a frustrated command purely based on its syntactic placement.
Coded and Evolving Meanings
Contextualization also allows models to identify hidden or altered meanings. Zhu et al. leveraged context-aware language models for euphemism detection, demonstrating that the model can isolate when an innocuous word is being used in an illicit or coded context by analyzing whether its surrounding environment matches its typical usage.
Downstream Impact
The transition from static to dynamic representations fundamentally changed how downstream applications process text. Instead of relying on rigid keyword matching, systems can perform deep semantic alignment.
- Information Retrieval: Dai et al. showed that contextual models revolutionize search relevance by using stopwords and punctuation to disambiguate terms, allowing for precise semantic matching based on query intent rather than raw term frequency.
- Dialogue Evaluation: Ghazarian et al. integrated contextualized representations into chatbot evaluation, proving that tracking how word meanings shift across an entire conversation history aligns automatic metrics much closer to human judgment.
- Phrase Retrieval: Moving beyond single words, Cai et al. demonstrated that representing a multi-word phrase based on its specific occurrence—rather than an average of all its historical uses—is critical for accurate cross-lingual phrase retrieval and machine translation.
By treating word representation as a dynamic reflection of context rather than a static definition, models mirror the flexible, adaptive way humans construct meaning.