Modeling Semantic Evolution with Diachronic Word Embeddings

74 concepts63 papers2 proposals

In computational linguistics, the foundational principle is that "you shall know a word by the company it keeps." As Brunila et al. and LaViolette note, this distributional view allows us to represent words as vectors in a high-dimensional space. However, language is not static. The "company" a word keeps changes as society evolves, technology advances, and political climates shift. Diachronic word embeddings extend traditional models by adding a temporal dimension, allowing researchers to track how the meaning of a word "drifts" through semantic space over years, decades, or centuries.

The Challenge of Alignment

The simplest way to track change is to divide a corpus into discrete temporal snapshots—such as decades or years—and train a separate embedding model for each. Abercrombie et al. and Batista-Navarro use this approach to capture the distributional properties of language in UK Parliamentary debates across a century.

However, this creates a fundamental mathematical hurdle: the alignment problem. Because word embedding algorithms (like Word2Vec) initialize vectors randomly, two models trained on different data will result in coordinate systems that are rotated or flipped relative to one another. You cannot simply compare the vector for "broadcast" in 1920 to "broadcast" in 2020 because they exist in different latent spaces.

To solve this, Yao et al. propose a joint learning framework. Instead of training models independently and aligning them later, they factorize matrices across all time slices simultaneously, using a smoothing regularization term to ensure that the latent space remains consistent while still allowing individual word vectors to move. This ensures that any measured distance between time periods reflects actual semantic shift rather than random initialization noise.

From Snapshots to Continuous Trajectories

While discrete bins are useful for long-term historical analysis, they can miss rapid, "bursty" changes. Kutuzov et al. demonstrate that by using incrementally updated embeddings, models can detect "cultural shifts"—such as the changing associations of geographical locations during armed conflicts—much faster than standard linguistic drift.

Taking this further, Rosenfeld et al. and Erk introduced DiffTime, a model that treats time as a continuous variable. By making the neural network differentiable with respect to time, the authors can model smooth semantic trajectories. This allows researchers to use calculus to calculate the "velocity" of a word's meaning change at any specific point in history.

Other researchers treat semantic evolution as a sequential modeling task. Tsakalidis et al. and Liakata use LSTMs to model the entire path of a word's vector. By viewing the trajectory as a sequence, they can frame semantic shift as an anomaly detection problem, identifying moments where a word’s usage deviates sharply from its historical trend.

Contextualized and Decontextualized Shifts

Modern Transformer-based models like BERT provide a new lens for diachronic analysis. Unlike static vectors, these models generate different embeddings for the same word depending on its sentence context.

Rodina et al. show that by averaging these contextual embeddings into "word prototypes" for different eras, they can effectively mirror human judgments of how Russian word meanings shifted from the pre-Soviet to the post-Soviet era. Similarly, Spliethöver et al. argue that these "decontextualized" embeddings provide a more stable window into social biases in media than traditional methods, which often suffer from data sparsity in older texts.

For even finer granularity, Kishino et al. and Yamagiwa apply Unbalanced Optimal Transport to contextualized embeddings. Their method, Sense Usage Shift (SUS), quantifies semantic change at the level of individual sentences, allowing researchers to see exactly which specific senses of a word are emerging or dying out.

Applications: Mirroring Society

The power of diachronic embeddings lies in their ability to quantify social history. Garg et al. used decade-specific embeddings to show how gender and ethnic stereotypes in the US have evolved over 100 years, finding that the movement of word vectors closely mirrors real-world changes in demographics and occupations.

These models also serve as predictive tools. Kutuzov et al. found that by tracking how the relationships between armed groups and locations shift in news coverage, they could predict future conflict participants. In the realm of public discourse, Reddy et al. used monthly Word2Vec models to track the rise of Islamophobia on Twitter during the COVID-19 pandemic, illustrating how global crises can trigger rapid shifts in hateful rhetoric.

By treating language as a dynamic geometric system, diachronic embeddings provide a scalable, quantitative bridge between the fields of natural language processing and the social sciences, as argued by Jacob Eisenstein. They transform static text archives into a living map of human thought and cultural evolution.

Key Foundational Work

Multiscale topic tomography (2007) — contributed 1 concept elements to this direction.
Temporal Classification of Text and Automatic Document Dating (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_diachronic_word_embeddings_3110b2c4,
  title = {diachronic word embeddings},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/diachronic-word-embeddings/rdr_3110b2c49f1d49f28262b5862437c076},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_diachronic_word_embeddings_3110b2c4_v1,
  title = {diachronic word embeddings},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/diachronic-word-embeddings/rdr_3110b2c49f1d49f28262b5862437c076/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake