To track how language evolves over decades or centuries, computational linguists frequently rely on word embeddings. If we want to see how the word "apple" shifted from a crisp fruit to a technology giant, or how "broadcast" moved from an agricultural method of scattering seeds to a media phenomenon, a natural starting point is to train one embedding model on historical text and another on modern text.
However, a fundamental mathematical barrier arises: embeddings generated from separate, time-stamped corpora do not naturally share a coordinate system (On the Linearity of Semantic Change: Investigating Meaning Variation via Dynamic Graph Models; A Word Sense Distribution-based approach for Semantic Change Prediction). Without a shared reference frame, calculating simple arithmetic differences to measure semantic shifts over time is impossible.
The Geometry of Misalignment
Why do these vector spaces misalign? Word embedding algorithms are inherently stochastic (Diachronic word embeddings and semantic shifts: a survey). Because the underlying optimization landscapes are non-convex, training models independently on different data subsets results in inconsistent vector space orientations (Dynamic Word Embeddings; Temporal dynamics of semantic relations in word embeddings: an application to predicting armed conflict participants).
Imagine two globes. On both globes, London and New York are the exact same distance apart. However, if the second globe is rotated upside down, the absolute latitude and longitude coordinates for New York will be completely different.
This is exactly what happens in high-dimensional vector spaces. Even if the internal semantic relationships between words remain identical across eras, one space might be rotated, scaled, or reflected relative to another. This lack of alignment prevents researchers from distinguishing between actual shifts in word usage and random variations in the model's coordinate system (Visualizing Trends of Key Roles in News Articles; Apprentissage de plongements de mots dynamiques avec régularisation de la dérive (Learning dynamic word embeddings with drift regularisation)).
Post-Hoc Alignment and the Anchor Dilemma
The traditional way to solve this is to train separate models for distinct time intervals and then map them into a common coordinate system using a mathematical transformation, such as an orthogonal Procrustes rotation (Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change). To calculate this transformation matrix, researchers select a set of "landmark" or "anchor" words—terms assumed to have stable meanings across eras (like "house" or "water").
Independent Spaces (Misaligned) Post-Hoc Alignment via Anchors
Time 1 Time 2 Aligned Space
[ apple ] ( ) [ apple ]
[ chair ] ( chair ) ===[Rotation]===> [ chair ] ( chair )
[ river ] ( river ) [ river ] ( river )
( apple ) ( apple )
However, this approach introduces a subtle catch-22. As Tsakalidis et al. demonstrate in Mining the UK Web Archive for Semantic Change Detection., including words that are currently undergoing semantic shifts in the alignment set introduces noise. If an anchor word's meaning has actually changed slightly, forcing it to align distorts the entire transformation matrix, effectively masking the very semantic changes the analysis aims to detect.
Gruppi et al. similarly note that the number and choice of landmark words heavily impact performance; forcing shifting words into alignment artificially minimizes their distance, hiding their evolution (SChME at SemEval-2020 Task 1: A Model Ensemble for Detecting Lexical Semantic Change). Furthermore, Dubossarsky et al. warn that the mathematical alignment process itself can introduce artifacts that mimic semantic shifts, making it difficult to distinguish between genuine linguistic evolution and errors generated by the transformation (Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change).
Implicit Alignment via Shared Compasses
To bypass the pitfalls of post-hoc rotation, alternative frameworks force alignment during the training process. Instead of allowing spaces to drift randomly and correcting them later, these methods anchor the models to a shared foundation from the start.
One prominent approach is the Temporal Word Embeddings with a Compass (TWEC) framework introduced by Bianchi et al. (Training Temporal Word Embeddings with a Compass). Instead of training time slices independently, the authors propose training a static, "atemporal" model on the entire corpus to act as a frozen reference matrix—the compass. Time-specific models are then trained by optimizing against this shared target matrix. This implicitly aligns the resulting vector spaces, balancing the capture of semantic shifts with the maintenance of a stable global structure.
Implicit Alignment (TWEC)
[ Atemporal Compass Matrix ] (Fixed Reference)
/ \
v v
[Time 1 Embeddings] [Time 2 Embeddings]
(Naturally Aligned) (Naturally Aligned)
A continuous alternative, proposed by Bamler et al., treats word vectors as latent trajectories governed by a diffusion process, akin to a Kalman filter (Dynamic Word Embeddings). By modeling the transition between sequential time steps as a smooth random walk, the architecture naturally maintains alignment across slices, bypassing data sparsity issues in specific eras.
Similarly, terms can be mapped across eras by analyzing shared contextual "bridge terms" rather than requiring direct co-occurrence, allowing systems to link modern concepts to historical counterparts that never appeared in the same documents (Bridging the Terminology Gap in Web Archive Search).
Modeling the Trajectory of Meaning
Once a shared semantic space is established, researchers can track trajectories across various granularities—from rapid online sentiment shifts over months (On Xing Tian and the Perseverance of Anti-China Sentiment Online) to slow, macro-level changes in valence and dominance over centuries (CHRONOBERG: Capturing Language Evolution and Temporal Awareness in Foundation Models).
While early work explored whether semantic change follows predictable linear patterns (On the Linearity of Semantic Change: Investigating Meaning Variation via Dynamic Graph Models), language evolution is frequently non-linear. To capture these complex transitions, Tsakalidis et al. utilized deep learning architectures, such as LSTM-based encoder-decoders, to model the sequential trajectories of word embeddings (Sequential Modelling of the Evolution of Word Representations for Semantic Change Detection). By treating semantic shift as an anomaly detection problem along a continuous path, these models successfully maintain the identity of words across historical periods while sensing the subtle, non-linear shifts of their eras (Time-Aware Language Modeling for Historical Text Dating).
Key Foundational Work
Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change (2016) — contributed 1 concept elements to this direction.
On the Linearity of Semantic Change: Investigating Meaning Variation via Dynamic Graph Models (2016) — contributed 1 concept elements to this direction.
Bridging the Terminology Gap in Web Archive Search (2009) — contributed 1 concept elements to this direction.