Neighborhood-Coupled Dynamic Word Embeddings: Testing Whether Meaning Change Spreads Between Regions

Methods that track how word meanings change over time almost always treat a language as one undivided text stream, or as a set of separate corpora that never influence each other. Epidemiology and regional forecasting have shown repeatedly that this "spatial independence" assumption is costly when the thing being modeled actually travels between places. Yet the tools for semantic change have no way to say that a new sense of a word appeared in one city and arrived in another a year later. We propose a dynamic word embedding model in which each region has its own word vectors, and each region's vectors at the next time step are pulled toward the earlier vectors of the regions that influence it. The strength and direction of that pull are learned from text alone.

The bet is concrete. If meaning change really diffuses between places, a model that lets a region "look at" its neighbors' past should predict that region's future text better than a model that only looks at its own past. It should also beat a model that simply smooths all regions toward each other at the same moment. The influence pattern it learns should point in the right direction when a sense's origin is known. If none of this holds, that is also informative: it would mean regional semantic variation is mostly static offsets, not diffusion. The experiments are designed so that either outcome is clean.

Core Claim

A word embedding is a vector per word learned from which words appear near it. A dynamic embedding lets that vector drift over time. Dynamic Word Embeddings does this by treating each word's vector as a random walk, where the next time step is the current vector plus Gaussian noise. That prior shares information across time slices, and it is why their dynamic models kept good held-out likelihood on small corpora where per-slice static models "often failed entirely." We extend the random walk across space. Let $w_{r,t}$ be a word's vector in region $r$ at time $t$, let $B_{rj} \ge 0$ be the learned influence of region $j$ on region $r$ (sparse and directed, so $B_{rj} \ne B_{jr}$ in general), and let $D$ be the diffusion variance. The prior is

$$ w_{r,t+1} \sim \mathcal{N}\Big(w_{r,t} + \sum_{j \ne r} B_{rj}\,(w_{j,t} - w_{r,t}),\; D\,I\Big). $$

The right-hand side is the network autoregressive form that Armbruster uses for regional time series: own past plus weighted neighbors' past. Here it is lifted from scalars to embedding trajectories. Setting $B = 0$ recovers independent regional dynamic embeddings. A fixed symmetric $B$ applied at the same time step amounts to Laplacian smoothing between connected units, as in Socialized Word Embeddings. The new ingredient is the lagged, directed coupling estimated from data.

This also handles the alignment problem. Separately trained embeddings live in arbitrarily rotated coordinate systems, so vectors from different models cannot be compared directly. We let context vectors be shared across regions at each time, while they still drift over time as in the original model. Only the target-word vectors are regional, so every region is expressed in the same coordinates by construction, and no Procrustes rotation (the post-hoc fitting of one embedding space onto another) is needed between regions.

Relation to Prior Work

Three lines of work motivate the gap. A Latent Variable Model for Geographic Lexical Variation models regional word use from GPS-tagged tweets but uses a single month and explicitly leaves temporal dynamics unmodeled. Geographic Adaptation of Pretrained Language Models makes a language model's internal space mirror physical distance, but it is a static snapshot. Tracing Semantic Variation in Slang spans two centuries but reduces geography to a US/UK label and predicts origin from semantic neighborhoods, not from any cross-region transmission. All three treat place as a fixed attribute. We treat it as a node in a network that information flows through.

The closest network work is Semantic shift in social networks. It relates semantic change in 45 subreddits to community structure, but it analyzes each community independently against a generic corpus over two slices, so it never models flow between communities. From Paris to Berlin supplies the spirit of our method: it finds directed city-to-city influence by Granger causality on popularity curves, and it finds interaction modeling beats isolated analysis. However, it does this for frequencies of discrete visual style clusters. We estimate influence inside a continuous meaning space, where the question is not "did usage of X rise" but "did X start to mean something new here after it did there."

Two papers constrain the design. SemEval-2020 Task 1 found that nearly all top systems used static skip-gram embeddings rather than contextual transformers, and that systems over-predict change for low-frequency words. We therefore build on type-level skip-gram embeddings and control for frequency explicitly. Confounds and Consequences in Geotagged Twitter Data shows that GPS-tagged users skew younger than users who give a profile location (expected age 36.17 vs. 38.35) and use more slang. An apparent spread of meaning from one region to another could therefore be a shift in who is posting. We build that check into the evaluation rather than assuming it away.

Research Program

Inference. We fit the model by alternating two steps. With $B$ fixed, we run variational smoothing over all regions and times. The spatial coupling breaks the tridiagonal structure that gives the original model linear-time inference, but it leaves a block-banded structure that remains tractable for tens of regions at modest dimension (around 100). With trajectories fixed, $B$ is re-estimated as a sparse non-negative regression of each region's step $w_{r,t+1} - w_{r,t}$ on lagged neighbor differences, pooled over a fixed vocabulary of a few thousand words. We fit both a single global $B$ and a variant with one $B$ per coarse semantic field. The second tests whether slang and, say, technology terms travel along different routes.

Corpora. The model needs a multi-year, region-tagged corpus, and nothing in our file provides one at fine spatial resolution. The one-month Twitter sample of Eisenstein et al. is too short. We will therefore assemble a multi-year geotagged social-media corpus over US metropolitan areas or states, keeping GPS-tagged and profile-location subsets separate following Pavalanathan and Eisenstein's protocol.

We also build a second, historical two-node setting. We train on American and British Google Ngrams, which supply regional text over time, and evaluate against Green's Dictionary of Slang dates and regions of sense emergence. Two nodes is a crude network. Its value is that it has ground-truth direction.

Analyses. We will report three kinds of result. The first is whether coupling improves forecasts of future regional text. The second is whether the learned $B$ correlates with geographic distance and, where available, mobility, without ever being given them. The third is whether the regions identified as leaders change once demographic composition and frequency are controlled for.

Baselines

The naive reference is a global dynamic skip-gram model fit on all regions pooled. It has no regional vectors and represents the field's default. Every regional model must beat it on region-specific held-out likelihood, or regionalization adds nothing.

The independence baseline is the same model with $B = 0$. Each region gets its own random walk, with context vectors still shared so the comparison isolates coupling rather than alignment. This is the direct analogue of the univariate silos that Armbruster's network models beat. It uses identical data, dimension, vocabulary, and inference budget.

The strongest competitor is a pooling model in which regional vectors are a global dynamic vector plus a regional offset. The offsets are tied by a symmetric geographic Laplacian at the same time step, in the style of Socialized Word Embeddings with a time dimension added. A second pooling variant pulls each region toward the global mean, which plays the role of the coherence loss of Al-Halah and Grauman. These models capture "nearby places talk alike" without any notion of who leads. They are the real threat, because most of the likelihood gain from coupling could come from sharing statistical strength across regions rather than from diffusion. The pooling models get the same hyperparameter search budget as ours: the same grid over $D$, regularization strength, and dimension, selected on a validation slice.

The key ablation keeps everything else fixed and swaps the lagged directed $B$ for a contemporaneous symmetric $B$ with the same number of free parameters. Beating the pooling models means a statistically reliable gain in next-slice held-out log-likelihood that is concentrated in regions the model labels as followers. A uniform gain everywhere would point to pooling, not diffusion.

For origin attribution on the slang data, the reference point is the Exemplar chaining model of Sun and Xu, which reports roughly 59% mean accuracy on US/UK origin prediction against a 50% random baseline. Their task predicts origin from prior semantic history, whereas ours attributes origin from text up to a few years after emergence. We will therefore run their Exemplar model on our exact sense split and time cutoff, and treat 59% as a reference, not a head-to-head number. For held-out log-likelihood there is no reported result found on Lacuna for any spatially resolved corpus, so all likelihood baselines are run by us.

Why not just…?

Why not just train independent embeddings per region and run Granger tests on their per-word change scores afterwards? This two-stage pipeline is the most natural cheap alternative, and we include it. Its weakness is that slicing by both region and time makes each slice small. Bamler and Mandt show that independently trained slices break down exactly in that regime, and the resulting alignment noise would masquerade as lead-lag signal. Our planted-shift experiment (below) measures directly whether the two-stage pipeline recovers a known propagation graph as reliably as the joint model. If it does, the joint model is unnecessary, and we will say so.

Why not just track how word frequencies spread between regions? For brand-new words, frequency spread may be all one needs. But most meaning change happens to existing words whose counts barely move. Cultural Shift or Linguistic Drift? documents large shifts in words like cell and gay that frequency does not capture, and Sun and Xu found semantic chaining beat regional frequency for slang origin. We still fit a network autoregression on regional frequencies as a baseline. We then report results separately for words with stable versus changing frequency. If frequency diffusion matches us on the stable-frequency words, our claim fails.

Why not just use contextual embeddings with unbalanced optimal transport between region pairs? Quantifying Lexical Semantic Shift via Unbalanced Optimal Transport is a method that matches individual word-use vectors between two corpora while allowing senses to appear or vanish. It reaches 0.83 sense-level correlation with human judgments and separates broadening from narrowing, which our type-level vectors cannot. It is, however, a pairwise two-corpus comparison with no temporal model. Running it on every region pair at every time step gives descriptive shift scores but no forecast and no propagation structure, and SemEval-2020 cautions that contextual models did not dominate static ones for change detection. We view it as a complementary measurement layer. As a robustness check, we test whether our learned leader-follower pairs also show larger excess transported mass in the follower region one step later.

Proposed Experiments

The first decisive check is a planted-diffusion test on real text. In the multi-year geotagged corpus, we pick rare pseudo-words, or substitute one word into another word's contexts, and inject an artificial sense shift that starts in one region. The shift then spreads along a known directed graph with known delays. Because the ground truth is fully controlled while the background text is real, we can measure graph recovery (precision and recall of directed edges) and lag recovery for our model, the two-stage Granger pipeline, and the pooling model. The pooling model should be unable to recover direction at all. If our model cannot recover planted direction at realistic shift sizes, nothing it says about real data should be trusted.

The second check is forecasting. We train on years 1 through $T-1$ and score held-out log-likelihood of year $T$ text per region, against all baselines above. The prediction to test is that the gain over the pooling model concentrates in follower regions and in semantic fields with high estimated coupling.

The third check concerns direction against ground truth. In the US/UK Ngrams setting, for each Green's Dictionary sense with known region and date, we ask whether the learned per-field coupling and the order of shift onset point from the documented origin to the other region. We compare against the Exemplar model rerun on our split.

The fourth check targets confounds. We refit the geotagged model separately on the GPS-tagged and profile-location subsets. Leader-follower edges that appear in only one subset are flagged as likely demographic artifacts. All per-word shift statistics are computed both raw and as the shuffled-null "rectified change" of Shoemark et al., to remove the frequency bias SemEval-2020 reported.

The proposal stands if the planted test recovers direction, the forecasting gain survives the pooling baseline, and the leader regions persist across both sampling subsets. It is refuted, usefully, if pooling explains the gains.