Semantic Centroids from Contextualized Word Representations

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 150 papers60 conceptsfrozen 2026-10-04

In the early 1990s, Hinrich Schütze introduced the concept of Word Space in Word Space, suggesting that a word's meaning could be induced by aggregating the feature vectors of all its occurrences across a corpus. This distributional approach—averaging the various contexts in which a word appears—created a single vector reflecting its overall profile.

Decades later, the rise of Transformers like BERT and GPT shifted the field toward dynamic, contextualized embeddings where a word's vector changes based on its neighbors. However, researchers have discovered that these dynamic models still harbor a "semantic core" that can be recovered through a process often called decontextualization or contextual aggregation.

The Intuition of Marginalization

A contextualized model represents the word "bank" differently in the phrases "river bank" and "investment bank." If you want to know what the model "thinks" about the concept of a bank in general, looking at a single instance is insufficient.

The core mechanism of this research direction is to sample a diverse set of sentences containing a target word, extract the hidden state vectors for that word from a pre-trained model, and calculate their centroid (the mean vector). As Rishi Bommasani and Kelly Davis argue in Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings, this parameter-free reduction effectively "marginalizes out" the noise of specific sentence structures. By averaging across many contexts, the idiosyncratic "flavors" of individual sentences cancel each other out, leaving behind a robust, stable representation of the word’s lexical meaning.

Why Averaging Beats Isolation

One might assume that the simplest way to get a "static" vector from BERT is to feed the model the word in isolation. However, Ivan Vulić and colleagues have repeatedly shown—in works like Probing Pretrained Language Models for Lexical Semantics and On Cross-Lingual Retrieval with Multilingual Text Encoders—that this is suboptimal.

Modern Transformers are trained on natural language, not dictionary lists. A word in isolation is "out of distribution" for the model. By averaging embeddings from real-world usage, researchers capture the lexical prototype. Felix Hill posits in Why transformers are obviously good models of language that the model's input weights themselves function as these prototypes, representing the statistical average of all training contexts.

Applications Across Domains

This "averaging" strategy has proven surprisingly effective for tasks where stability is more important than context-sensitivity:

  • Social Bias Measurement: Maximilian Spliethöver and Maximilian Keiff found in No Word Embedding Model Is Perfect that decontextualized embeddings provide a more accurate window into underlying media prejudices than traditional static models, as they combine the semantic depth of Transformers with the interpretability of word-association tests.
  • Scientific Discovery: In From Tokens to Materials, Nan Wu and Wenjie Zhang use "contextual entity aggregation" to create a representational fingerprint of materials. By averaging embeddings from thousands of scientific papers, they synthesize distributed knowledge that a single formula cannot capture.
  • Psycholinguistics: Federico Bianchi and Fritz Guenther demonstrate in Meaning Modulations and Stability in Large Language Models that these averaged prototypes align more closely with human similarity ratings than isolated word vectors, suggesting that human semantic memory might operate on a similar principle of contextual integration.

Geometric Linearity

Perhaps the most striking finding in this area is the discovery of hidden linear structures. Matthew Trager and Alessandro Achille show in Linear Spaces of Meanings that "ideal word" vectors—calculated by averaging across diverse contexts—allow complex concepts to be reconstructed through simple vector addition. This suggests that while individual activations are messy and context-dependent, the model’s internal "map" of concepts is remarkably structured and accessible through simple geometric averaging.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_centroids_from_contextualized_w_509f3be4,
  title = {Semantic Centroids from Contextualized Word Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-centroids-from-contextualized-word-representations/rdr_509f3be4b7184248bcd367ff0a65bba1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_centroids_from_contextualized_w_509f3be4_v1,
  title = {Semantic Centroids from Contextualized Word Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-centroids-from-contextualized-word-representations/rdr_509f3be4b7184248bcd367ff0a65bba1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}