Inducing Word Senses by Clustering Contextual Representations

119 concepts95 papers1 proposal

Language is inherently ambiguous. A single word like "bank" can mean a financial institution, the edge of a river, or a banking shot in billiards. Traditionally, Natural Language Processing (NLP) relied on hand-crafted dictionaries like WordNet to catalog these meanings. However, manual annotation is expensive, language evolves rapidly, and rigid taxonomies often fail to capture task-specific nuances. As Lee demonstrated in Optimizing Classifier Performance in Word Sense Disambiguation by Redefining Sense Classes, task-oriented sense classes discovered automatically from text can actually outperform human-designed dictionary structures in machine learning tasks.

The core idea behind Unsupervised Word Sense Induction (WSI) is simple: words with different meanings appear in different contextual environments. By representing these contexts geometrically and grouping them using clustering algorithms, we can discover word senses directly from raw text without any human labels.

The Geometric Intuition: From Word Space to Contextual Embeddings

The foundation of this approach dates back to the concept of "Word Space" introduced by Schütze et al. in Word Space. The authors showed that by clustering high-dimensional context vectors—built from the statistical distribution of surrounding words—distinct meanings naturally emerge as spatial groupings. Early methods used lexical features and traditional clustering techniques like McQuitty's Similarity Analysis (Purandare in Discriminating Among Word Senses Using McQuitty’s Similarity Analysis) or Expectation-Maximization with rich linguistic features (Chen et al. in Chinese Verb Sense Discrimination Using an EM Clustering Model with Rich Linguistic Features).

As neural networks advanced, researchers realized that static word embeddings (like Word2Vec or fastText) collapse all meanings of a word into a single vector, creating an "Achilles heel" for polysemous words. To fix this, Huang et al. in Improving Word Representations via Global Context and Multiple Word Prototypes proposed clustering context vectors to learn multiple distinct "prototypes" per word.

The real breakthrough came with transformer-based contextualized language models (like BERT and ELMo). Because these models generate a unique vector for a word in its specific context, the embedding space itself implicitly handles sense disambiguation. As Ribeiro et al. noted in L2F/INESC-ID at SemEval-2019 Task 2: Unsupervised Lexical Semantic Frame Induction using Contextualized Word Representations, this implicit disambiguation allows simple "hard" clustering algorithms to succeed where complex fuzzy clustering was once required.

The Mechanics of Clustering Contextual Spaces

How exactly do these senses separate in latent space? When a large language model processes a corpus, instances of a polysemous word are mapped into high-dimensional space. If we visualize this space, tokens used in a financial context cluster together, while tokens used in a geographical context form a separate group.

Different clustering algorithms offer unique advantages for capturing these distributions:

Bridging Latent Clusters and Structured Knowledge

While discovering clusters is useful, downstream applications often need to map these latent groups back to human-readable definitions or structured knowledge bases.

To bridge this gap, Scarlini et al. developed the ARES framework in With More Contexts Comes Better Performance: Contextualized Sense Embeddings for All-Round Word Sense Disambiguation. ARES clusters contextualized embeddings from raw text and uses graph-based algorithms to map those clusters directly to WordNet senses, creating high-quality sense embeddings without manual annotation. Similarly, Mizuki et al. in Semantic Specialization for Knowledge-based Word Sense Disambiguation used an unsupervised alignment strategy with pseudo-labels to minimize the distance between a word's context embedding and its predicted sense embedding, fine-tuning the underlying model to better align with structured sense inventories.

Applications Across NLP Domains

The ability to induce senses without labels has unlocked solutions to diverse linguistic and practical challenges:

Key Foundational Work

Word Space (1992) — contributed 2 concept elements to this direction.
Discriminating Among Word Senses Using McQuitty’s Similarity Analysis (2003) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 93 papers
  2. Version 1 · 2026-10-04 · 59 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_unsupervised_sense_induction_a85e87ae,
  title = {unsupervised sense induction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unsupervised-sense-induction/rdr_a85e87ae411c45e1a86808ea1a87672b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_unsupervised_sense_induction_a85e87ae_v2,
  title = {unsupervised sense induction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unsupervised-sense-induction/rdr_a85e87ae411c45e1a86808ea1a87672b/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake