Nonparametric Word Sense Induction

81 concepts54 papers3 proposals

When we look up a word in a dictionary, we expect to see a list of its various meanings. Some words are straightforward and monosemous, while others, like "run" or "bank," possess dozens of subtle nuances. Traditional natural language processing models, however, often compress a word's meaning into a single vector or force it into a rigid, pre-defined number of clusters ($k$).

As Paul Cook et al. point out, this fixed-parameter approach fails because the number of meanings varies drastically across a vocabulary. Forcing a static number of senses across all words causes models to over-split simple terms or under-split highly polysemous ones (Do Kook Choe et al.; Jordan Lee Boyd-Graber et al.). To capture natural language accurately, a model's representational complexity must scale dynamically based on the data density and context diversity of each individual word. This insight drives the field of nonparametric sense discovery.


Bayesian Nonparametrics and Infinite Capacity

How do we let the data decide the number of senses? One powerful framework is Bayesian nonparametrics, which treats the underlying sense space as potentially infinite. Instead of picking a fixed $k$ upfront, these models use stochastic processes to allocate probability mass to new senses only when the observed context diverges significantly from what has already been seen.

Consider the AdaGram (Adaptive Skip-gram) model proposed by Sergey Bartunov et al. It extends the standard Skip-gram architecture using a stick-breaking process—a constructive realization of a Dirichlet Process.

Imagine a stick of probability mass equal to 1. When the model encounters a word in a context, it evaluates how well the context fits existing sense prototypes. If the context is entirely novel, the model "breaks off" a piece of the remaining stick to initialize a brand-new sense prototype:

$$\text{Probability of a new sense} \propto \frac{\alpha}{n - 1 + \alpha}$$

where $\alpha$ is a concentration parameter controlling the tendency to spawn new clusters, and $n$ is the number of observed instances. This mathematical formulation allows frequent, highly ambiguous words to naturally acquire more semantic prototypes over time, while rare or unambiguous terms remain unified under a single representation.

Fixed-k Approach (e.g., k=3):
[ Word ] ---> [ Sense 1 ] [ Sense 2 ] [ Sense 3 ]  (Forced allocation)

Nonparametric Stick-Breaking Approach:
[ Word ] ---> [ Sense 1 ] 
         ---> [ Sense 2 ] 
         ---> [ New Context? ] ---> [ Break off new piece -> Sense 3 ]

A similar logic applies to the Hierarchical Dirichlet Process (HDP) utilized by Xuchen Yao et al. By framing senses as "dishes" shared across different "restaurants" (word contexts), the HDP automatically infers the optimal subset of senses directly from the data distribution. This generative flexibility allows models to dynamically discover an open-ended number of senses and hidden concepts without manual parameter tuning (Dirk Hovy et al.; Baobao Chang et al.).


Density-Driven Clustering in Embedding Spaces

With the rise of contextualized word representations like BERT and ELMo, the problem shifts from learning generative prototypes from scratch to clustering dense vector clouds. Because contextualized models output a unique vector for every single token occurrence, a word appears as a cloud of points in a high-dimensional space. The goal is to partition this cloud into discrete senses without pre-specifying the cluster count.

To achieve this, researchers leverage nonparametric clustering algorithms like Affinity Propagation. Unlike $k$-means, which requires an explicit cluster count and initializes arbitrary centroids, Affinity Propagation treats all data points as potential "exemplars." It passes real-valued messages between data points until a high-quality set of exemplars emerges based on the natural density of the data.

Domagoj Alagic demonstrated that applying affinity propagation to lexical substitutes allows the number of induced senses to emerge dynamically from the data distribution. When applied directly to contextualized embeddings, this density-driven strategy captures subtle, fine-grained variations in usage across different sectors or domains that fixed-$k$ methods routinely obscure (Alexandre Allauzen et al.; Matej Martinc et al.).

Other variations use graph-based nonparametric algorithms, such as Chinese Whispers, where word instances are represented as nodes in a graph, and clusters form organically through local neighborhood voting (Eugénio Ribeiro et al.). Alternatively, models can employ hierarchical clustering combined with distance thresholds or statistical criteria to determine exactly where to halt the merging process, ensuring the final representation adapts to the true level of polysemy (Zhaohui Wu 0002; Zhenzhong Zhang et al.).


Tracking Evolving Meanings Over Time

Language is a living system; word meanings continuously shift, emerge, and disappear. For instance, the word "apple" referred almost exclusively to fruit centuries ago, but it has since spawned a massive semantic cluster centered around technology. Parametric diachronic models that require a static, pre-defined number of latent senses fail to capture this fluid evolution (Lea Frermann et al.).

To solve this, Seiichi Inoue et al. and Daichi Mochihashi et al. introduced Infinite SCAN, a model designed to track diachronic semantic change without a fixed topic count. It achieves this by integrating a logistic stick-breaking process within a Markov Random Field (MRF).

Time Period 1: [Sense A: Fruit] (High probability mass)
Time Period 2: [Sense A: Fruit] ---> [Sense B: Technology] (Mass begins to shift)
Time Period 3: [Sense A: Fruit] ---> [Sense B: Technology] ---> [Sense C: Smart Device]

The MRF ensures temporal smoothness—meaning a word's senses in the year 1900 are statistically related to its senses in 1901—while the logistic stick-breaking process provides an infinite-dimensional representation space. When a novel usage pattern persists over a specific historical period, the model adaptively allocates probability mass to a brand-new sense.

By treating the number of senses as a learnable, time-varying variable, nonparametric diachronic models achieve significantly higher sense coherence. They allow the data itself to dictate when a meaning has truly drifted, preventing the omission of historical nuances and providing a cleaner window into how language evolves (Ashjan Alsulaimani et al.).

Key Foundational Work

Bayesian Semi-Supervised Chinese Word Segmentation for Statistical Machine Translation (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_non_parametric_sense_discovery_827296f2,
  title = {non-parametric sense discovery},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-parametric-sense-discovery/rdr_827296f2ed764a9282f910f7da04f41a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_non_parametric_sense_discovery_827296f2_v1,
  title = {non-parametric sense discovery},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-parametric-sense-discovery/rdr_827296f2ed764a9282f910f7da04f41a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake