In natural language, a single word often carries multiple, distinct meanings—a phenomenon known as polysemy. For decades, computational models struggled with this "representation crisis." Early word embeddings, like the original Word2Vec, assigned a single static vector to each word, effectively averaging the meaning of "bank" (the financial institution) and "bank" (the river edge) into a single, muddy point in space.
The research direction emerging from these papers focuses on Word Sense Induction (WSI): the fully automatic discovery of word meanings from raw text without relying on hand-coded dictionaries or human-labeled data. By leveraging the spatial properties of high-dimensional embeddings, researchers have found that different word senses naturally "clump" together, allowing machines to untangle polysemy through unsupervised clustering.
The Contextual Proxy for Meaning
The core intuition behind this field is that the meaning of a word is defined by its company. As early as 1992, Hinrich Schütze proposed the concept of Word Space, suggesting that if we represent the context of every occurrence of a word as a vector, the distinct meanings will emerge as natural clusters.
This idea evolved significantly with the advent of neural networks. Eric H. Huang, Richard Socher, and Andrew Y. Ng demonstrated in Improving Word Representations via Global Context and Multiple Word Prototypes that we can learn multiple "prototypes" for a single word. By clustering context vectors and relabeling word occurrences with their cluster ID, a model can learn specialized embeddings for different semantic roles.
From Static Vectors to Spatial Clusters
The breakthrough for WSI came with transformer-based models like BERT and RoBERTa. Unlike static embeddings, these models produce contextualized embeddings—unique vectors for every single instance of a word based on its specific sentence.
Researchers found that these contextualized vectors are not scattered randomly; they organize themselves into distinct spatial neighborhoods. Tommaso Pasini and colleagues introduced CluBERT, which uses this property to learn sense distributions across languages. Similarly, Gabriel Stanovsky and Mark Hopkins showed in their Odd-Man-Out task that clustering ELMo embeddings creates sense-specific prototypes that far outperform static models in capturing human-like word associations.
The mechanism is elegant in its simplicity:
- Collect all occurrences of a target word (e.g., "apple") from a large corpus.
- Generate a contextualized embedding for each occurrence.
- Cluster these embeddings using algorithms like K-Means or Affinity Propagation.
- Represent each sense as the centroid of its cluster.
Detecting Subtle Linguistic Shifts
This unsupervised approach is particularly powerful because it doesn't require a pre-defined list of meanings. This makes it ideal for detecting "weak signals" in language that dictionaries might miss.
- Semantic Change: Xiaohang Tang and Danushka Bollegala developed the Sense-based Semantic Change Score, which tracks how the "mix" of a word's senses shifts over time. Filip Miletic has applied similar clustering techniques to detect contact-induced semantic shifts in regional dialects, such as Quebec English.
- Pejorative Language: Liviu P. Dinu demonstrated that pejorative uses of words (using a term as a slur vs. a neutral description) can be separated as distinct senses in the latent space of BERT, even without labeled training data.
- Disciplinary Jargon: Jesse Dodge and David Bamman used WSI to show how common words take on secret, discipline-specific meanings in scholarly publications, acting as "gatekeepers" to knowledge.
Beyond Text: Acronyms and Audio
The utility of embedding clustering extends to specialized domains where dictionaries are often incomplete or outdated. In Acronym Disambiguation Using Word Embedding, Chao Li and Lei Ji showed that unsupervised embeddings can distinguish between "CMU" meaning Carnegie Mellon or Central Michigan by analyzing the surrounding semantic neighborhood.
Even in the realm of speech, Justin Chiu and Alan W. Black applied repeated bisection clustering to "utterance tokens" to induce word senses directly from spoken audio, proving that the statistical distribution of context is a robust signal across different modalities of language.
Why It Works
The success of these methods rests on the manifold hypothesis in NLP: that natural language usage occupies a low-dimensional manifold within the high-dimensional vector space. When a word is polysemous, its occurrences don't form a single blob; they form distinct sub-manifolds. By clustering, we are essentially performing a topological decomposition of a word's usage, allowing models to "see" the different facets of meaning that humans navigate intuitively.