Unsupervised Word Sense Induction from Contextual Embeddings

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 259 papers69 conceptsfrozen 2026-10-04

In natural language, a single word often carries multiple, distinct meanings—a phenomenon known as polysemy. For decades, computational models struggled with this "representation crisis." Early word embeddings, like the original Word2Vec, assigned a single static vector to each word, effectively averaging the meaning of "bank" (the financial institution) and "bank" (the river edge) into a single, muddy point in space.

The research direction emerging from these papers focuses on Word Sense Induction (WSI): the fully automatic discovery of word meanings from raw text without relying on hand-coded dictionaries or human-labeled data. By leveraging the spatial properties of high-dimensional embeddings, researchers have found that different word senses naturally "clump" together, allowing machines to untangle polysemy through unsupervised clustering.

The Contextual Proxy for Meaning

The core intuition behind this field is that the meaning of a word is defined by its company. As early as 1992, Hinrich Schütze proposed the concept of Word Space, suggesting that if we represent the context of every occurrence of a word as a vector, the distinct meanings will emerge as natural clusters.

This idea evolved significantly with the advent of neural networks. Eric H. Huang, Richard Socher, and Andrew Y. Ng demonstrated in Improving Word Representations via Global Context and Multiple Word Prototypes that we can learn multiple "prototypes" for a single word. By clustering context vectors and relabeling word occurrences with their cluster ID, a model can learn specialized embeddings for different semantic roles.

From Static Vectors to Spatial Clusters

The breakthrough for WSI came with transformer-based models like BERT and RoBERTa. Unlike static embeddings, these models produce contextualized embeddings—unique vectors for every single instance of a word based on its specific sentence.

Researchers found that these contextualized vectors are not scattered randomly; they organize themselves into distinct spatial neighborhoods. Tommaso Pasini and colleagues introduced CluBERT, which uses this property to learn sense distributions across languages. Similarly, Gabriel Stanovsky and Mark Hopkins showed in their Odd-Man-Out task that clustering ELMo embeddings creates sense-specific prototypes that far outperform static models in capturing human-like word associations.

The mechanism is elegant in its simplicity:

  1. Collect all occurrences of a target word (e.g., "apple") from a large corpus.
  2. Generate a contextualized embedding for each occurrence.
  3. Cluster these embeddings using algorithms like K-Means or Affinity Propagation.
  4. Represent each sense as the centroid of its cluster.

Detecting Subtle Linguistic Shifts

This unsupervised approach is particularly powerful because it doesn't require a pre-defined list of meanings. This makes it ideal for detecting "weak signals" in language that dictionaries might miss.

Beyond Text: Acronyms and Audio

The utility of embedding clustering extends to specialized domains where dictionaries are often incomplete or outdated. In Acronym Disambiguation Using Word Embedding, Chao Li and Lei Ji showed that unsupervised embeddings can distinguish between "CMU" meaning Carnegie Mellon or Central Michigan by analyzing the surrounding semantic neighborhood.

Even in the realm of speech, Justin Chiu and Alan W. Black applied repeated bisection clustering to "utterance tokens" to induce word senses directly from spoken audio, proving that the statistical distribution of context is a robust signal across different modalities of language.

Why It Works

The success of these methods rests on the manifold hypothesis in NLP: that natural language usage occupies a low-dimensional manifold within the high-dimensional vector space. When a word is polysemous, its occurrences don't form a single blob; they form distinct sub-manifolds. By clustering, we are essentially performing a topological decomposition of a word's usage, allowing models to "see" the different facets of meaning that humans navigate intuitively.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 93 papers
  2. Version 1 · 2026-10-04 · 59 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_unsupervised_sense_induction_a85e87ae,
  title = {unsupervised sense induction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unsupervised-sense-induction/rdr_a85e87ae411c45e1a86808ea1a87672b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unsupervised_word_sense_induction_from_c_a85e87ae_v1,
  title = {Unsupervised Word Sense Induction from Contextual Embeddings},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unsupervised-word-sense-induction-from-contextual-embeddings/rdr_a85e87ae411c45e1a86808ea1a87672b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}