When we think about representing the meaning of a word in context, our modern instinct is to reach for dense, high-dimensional contextualized embeddings like BERT or ELMo. These models look at a word's neighbors (syntagmatic relations) to compute a vector. However, there is an alternative, deeply intuitive way to capture meaning: instead of looking at what words surround a target, we look at what other words could replace it. This is the paradigmatic view of language.
For example, in the sentence "The financial institution collapsed," we can replace "institution" with "bank", "firm", or "company". In "The educational institution collapsed," the valid substitutes shift to "school", "university", or "college". The set of plausible substitutes acts as a highly specific semantic fingerprint. By representing a word instance through its distribution of potential lexical substitutes, we can build representations that are inherently interpretable, robust to contextual noise, and grounded in a shared vocabulary.
The Mechanism: Constructing a Bag of Substitutes
How do we systematically find these substitutes? Early approaches relied on searching massive n-gram corpora to find words that fit the same local context (Uzuner et al.; Schwartz et al.). Today, Masked Language Models (MLMs) provide a natural engine for this task. By masking the target word and querying the model's prediction head, we obtain a probability distribution over the entire vocabulary for that specific slot.
As Kudisov et al. and Eyal et al. demonstrate, this "bag-of-substitutes" (BOS) serves as a discrete, human-readable proxy for word sense. Instead of analyzing abstract vector spaces, a system can represent a word's meaning directly through its top-$k$ predicted substitutes.
Original Sentence: "The [target] was chewing on a piece of cheese."
Masked Context: "The [MASK] was chewing on a piece of cheese."
MLM Substitutes: [mouse: 0.65, rat: 0.20, hamster: 0.05, ... ] -> Sense: Animal
Original Sentence: "Plug the [target] into the USB port."
Masked Context: "Plug the [MASK] into the USB port."
MLM Substitutes: [device: 0.45, mouse: 0.30, keyboard: 0.10, ... ] -> Sense: Device
To ensure strict semantic equivalence, Lacerra et al. show that we can further filter these candidates by comparing the hidden state of the original target word with the hidden state of the substitute word placed into that same position, ensuring the replacement preserves the exact nuance required. Conversely, Kutuzov et al. demonstrate that we can map static type-level embeddings back into these contextual spaces to discover situational substitutes dynamically.
Why It Works: Isolating the Contextual Fingerprint
Standard contextual embeddings conflate the intrinsic meaning of the target word with the influence of its environment. Paradigmatic representations elegantly decouple them. By replacing the target word with a placeholder or evaluating alternative fillers, we isolate the exact semantic constraints imposed by the sentence.
Ebert et al. show that modeling the probability distribution of alternative words that could occupy a position creates a rich contextual fingerprint. This fingerprint is highly effective at resolving "hard" ambiguous words (such as distinguishing whether "hard" means "difficult" or "solid" based on its environment).
Similarly, Aina et al. leverage this by averaging the embeddings of valid alternative words to define a reference point for "pure context." This strategy proves that language models often remain heavily biased toward a word's static dictionary meaning; isolating the context via substitutes allows us to see exactly how much the environment is shifting the target's meaning. This context-awareness is critical for downstream tasks like text simplification (Paetzold et al.), paraphrase ranking (Apidianaki), and handling out-of-vocabulary words to preserve caption semantics (the authors).
Bypassing Vector Alignment in Semantic Change
One of the most compelling justifications for substitution-based representations is found in tracking how word meanings evolve over time (lexical semantic change). Traditionally, comparing contextual embeddings across different eras requires complex, lossy vector space alignment.
Card solved this by using an MLM's fixed vocabulary as a universal reference frame. Instead of comparing high-dimensional vectors across time periods, the method represents a word's meaning in an era by the aggregated probability distribution of its substitutes.
If the top substitutes for "broadcast" shift over a century from "scatter", "sow", and "disperse" to "transmit", "air", and "show", the semantic change is captured cleanly, efficiently, and with a fully interpretable "paper trail" that bypasses vector alignment entirely.
Driving Weak Supervision and Structure Learning
Because substitutability relies on structural and semantic interchangeability, it provides an excellent inductive bias for tasks with little to no labeled data.
Word Sense Induction
Alagic et al. demonstrate that clustering instances based on their lexical substitutes filters out irrelevant contextual noise far better than clustering raw co-occurrence patterns. The substitutes act as a semantic filter, discarding surface-level variations in the sentence and focusing purely on core meaning.
Weakly Supervised NER and Intent Detection
Peng et al. introduce a "Divergence Distance" metric for Named Entity Recognition. By swapping a candidate text span with a known seed entity (e.g., replacing an unknown phrase with a known entity like "Davinci"), they measure how much the surrounding context's probability distribution shifts. A low divergence proves that the candidate span and the seed entity are functionally interchangeable, allowing the system to mine high-quality training labels without manual annotation.
A similar compatibility framing is used by Gong et al. to identify central intentions in e-commerce queries by replacing candidate terms with placeholders to score their fit against the linguistic environment.
Grammar Induction
This principle extends even to syntax. van Zaanen highlights how identifying patterns of substitutability—where different word sequences can freely swap into the same environments—drastically constrains the search space for learning context-free grammars. If two sequences can fill the same slot, they likely share a structural category.
By shifting our perspective from what a word is to what a word can be, paradigmatic representations turn the vocabulary of a language model into its own meta-language for explaining context, structure, and meaning.
Key Foundational Work
Learning Syntactic Categories Using Paradigmatic Representations of Word Context (2012) — contributed 3 concept elements to this direction.
Improving Supervised Sense Disambiguation with Web-Scale Selectors (2012) — contributed 2 concept elements to this direction.
Word Sense Disambiguation for Information Retrieval (1999) — contributed 1 concept elements to this direction.