Resolving the Semantic Conflation Bottleneck in Word Embeddings

711 concepts487 papers1 proposal

When we represent a word as a single vector in a high-dimensional space, we make a fundamental assumption: that a single string of characters corresponds to a single, unified concept. For polysemous words—words with multiple distinct meanings—this assumption breaks down.

Consider the word apple. In a standard embedding space, it must simultaneously exist near pie, orchard, iPhone, and stock. As Wolf et al. observe, forcing a single point to span this entire semantic range effectively compels the model to average disparate contexts. Geometrically, this pull from competing semantic forces drags the vector into a compromised, noisy middle ground that accurately represents neither sense. Sun et al. point out that this conflation muddles downstream similarity measurements, while Zeng et al. describe a similar "semantic blurring" where literal and figurative expressions become mathematically indistinguishable.

Splitting the Point into Multi-Prototype Embeddings

The most intuitive escape from this bottleneck is to allocate multiple vectors per word type—an approach known as multi-prototype embedding. If a word has $K$ distinct meanings, it should possess $K$ distinct vectors in the embedding space.

But how do we determine $K$, and how do we assign contexts to the correct vector during training? Tian et al. address this by extending the standard Skip-Gram architecture into a probabilistic framework. They utilize an Expectation-Maximization (EM) algorithm to dynamically assign word occurrences to different latent senses based on their local context. To handle the structural variability of language, Zheng et al. introduce a regression model that dynamically estimates the number of active meanings a token possesses, allowing the architecture to scale its representation fluidly rather than relying on a fixed, hardcoded number of senses.

Grounding Senses with Structured Ontologies

While purely unsupervised clustering can separate distinct contexts, it often struggles with data sparsity or subtle semantic nuances. To solve this, a parallel line of research anchors vector spaces using external, human-curated knowledge bases like WordNet.

Instead of learning senses from scratch, Dodge et al. introduced "retrofitting," a post-processing technique that treats a semantic lexicon as a graph and iteratively nudges word vectors closer to their known relational neighbors. To bring this structural awareness directly into token representations, Dasigi et al. propose ontology-aware embeddings, where a word is represented as a context-sensitive weighted sum of explicit WordNet concepts. Similarly, Xu leverages a Convolutional Neural Network to compose WordNet definitions (glosses) directly into initial sense vectors, providing a semantically rich initialization that prevents distinct senses from collapsing into one another during training.

From Points to Regions and Subspaces

Representing word senses as discrete points—even multiple points—implies that a word's meaning is static within that specific sense. A more flexible alternative is to reimagine words not as points, but as continuous regions or geometric shapes.

Brazinskas et al. propose embedding words as probability distributions rather than fixed vectors. By predicting a density rather than a point, the model can express semantic uncertainty and breadth. However, the authors note a key challenge: the variance of a distribution can inadvertently conflate a word's frequency in the dataset with its actual semantic scope. To better capture multi-modal polysemy, Muzellec et al. expand this idea into the Wasserstein space of elliptical distributions, allowing multi-modal mixtures to represent coexisting meanings. Moving from probabilistic regions to linear algebra, Luo et al. argue that concepts are best modeled as low-dimensional subspaces rather than individual points, allowing the representation to naturally accommodate contextual variations without losing its core identity.

Dynamic Disambiguation and Contextual Routing

Ultimately, resolving the polysemy bottleneck requires downstream architectures to select or compose the correct meaning on the fly. This is frequently achieved by integrating global context or topic models directly into the neural composition layer.

Zhao demonstrates this by combining Latent Dirichlet Allocation (LDA) with deep compositional models, using global topic embeddings to dynamically disambiguate words during sentence classification. In a similar vein, Das et al. integrate Gaussian LDA with word vectors to ensure the model distinguishes between different semantic contexts. Even within large-scale pre-trained language models, surface-level prediction tasks can fail to capture explicit semantic boundaries. To address this, Levine et al. introduced SenseBERT, which adds a masked word-sense prediction task to the pre-training phase, forcing the network to explicitly resolve polysemy by predicting WordNet supersenses from context.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 511 papers
  2. Version 1 · 2026-10-04 · 526 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_polysemy_representation_gap_9507fd28,
  title = {polysemy representation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/polysemy-representation-gap/rdr_9507fd2855764df28acfddaff8eb571c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_polysemy_representation_gap_9507fd28_v2,
  title = {polysemy representation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/polysemy-representation-gap/rdr_9507fd2855764df28acfddaff8eb571c/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake