Does Frequency Really Protect Word Meaning? A Matched, Noise-Equalized Test of the Law of Conformity

A widely cited finding in computational linguistics says that common words change meaning more slowly than rare ones. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change fit this pattern as a power law across four languages and two centuries. A later negative-control study, Outta Control, found that much of the pattern also appears in a corpus whose time periods were shuffled, where no real change can exist. The reason is simple. A word's measured "change" is a distance between two vectors estimated from finite samples of text. Rare words give noisier estimates, so they look as though they moved even when they did not. Frequency is also tangled up with polysemy (how many senses a word has), part of speech, and how central the word is to its semantic category. The field therefore does not know how much of the frequency effect is real.

This note proposes a study design that removes both problems at once, and it is not simply "apply propensity score matching to words." Propensity score matching is a method from observational causal inference. It pairs treated and untreated units that had the same predicted probability of treatment given their background traits, so that the two groups look alike on those traits. Matching by itself cannot fix the key problem, because here the treatment (frequency) also degrades the measuring instrument. The design therefore adds two ingredients. The first equalizes measurement: every word's meaning is estimated from exactly the same number of occurrences in each period. The second matches high- and low-frequency words on pre-period polysemy, part of speech, prototypicality, and prior frequency trend. A shuffled-corpus negative control is then subtracted within the matched sample. What survives is an estimate of how much entrenchment through frequency protects meaning, with a known error model.

Core Claim

The observed relation between frequency and drift contains two separable components. One is a measurement component, driven by how many tokens the estimate is built from. The other is a linguistic component, driven by how entrenched the word is in the speech community. Only the second is what episodic-memory accounts of frequency predict, such as Pierrehumbert's argument in Phonological Representation: Beyond Abstract Versus Episodic that dense exemplar clouds resist new usages.

We claim the linguistic component can be identified with three steps: fix the token count $n$ used to estimate each word's representation, balance pre-period covariates by propensity score, and difference out any residual artifact using a chronologically shuffled copy of the same corpus. Let $T_w=1$ for words in a high baseline-frequency band and $T_w=0$ for a low band, restricted to the frequency range where the two bands overlap on covariates. Let $D_w(n)$ be the drift of word $w$ measured from exactly $n$ sampled occurrences per period. The target estimand is

$$\tau(n) = \big(\bar D^{\text{real}}_{T=1}(n) - \bar D^{\text{real}}_{T=0}(n)\big) - \big(\bar D^{\text{shuf}}_{T=1}(n) - \bar D^{\text{shuf}}_{T=0}(n)\big),$$

where bars denote means over the propensity-matched sample and "shuf" is the time-shuffled corpus. Holding $n$ fixed equalizes sampling variance by construction. The shuffled term absorbs whatever frequency-linked artifact remains, for example from how often a word's neighbors occur. A Law of Conformity that is real predicts $\tau(n)<0$, stable across $n$ and across drift metrics. A pure artifact predicts $\tau(n)\approx 0$.

The same design answers the polysemy dispute. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change reported that polysemy speeds up change. Outta Control reported that this effect vanishes once frequency is controlled. We swap the roles: polysemy becomes the treatment and frequency becomes a matched covariate.

Relation to Prior Work

Two papers motivate the study, and they disagree. Hamilton et al. used linear regression on Procrustes-aligned embeddings and reported that frequency and polysemy explain 48% to 88% of drift variance. Dubossarsky et al. showed analytically that expected cosine distance between two samples of the same word falls with sample size. They also showed that frequency still explains 56% of variance on a shuffled corpus, against 67% on the real one. The correlations on shuffled data were $r=-0.915$ for count vectors, $-0.793$ for SVD embeddings, and $-0.295$ for PPMI (positive pointwise mutual information) vectors. Their controls diagnose the artifact but do not produce a corrected effect size. Nor do they balance words on linguistic covariates. We use their shuffled corpus as one ingredient, not as the whole answer.

Cultural Shift or Linguistic Drift? supplies a constraint and a second outcome. Global displacement (how far a word's vector moves) and local neighborhood change (how much its nearest-neighbor list turns over) track different processes: the first tracks regular drift in verbs, the second cultural shocks in nouns. That paper controls frequency with linear mixed-effects models, which neither check overlap nor equalize measurement. We therefore report $\tau(n)$ separately for both metrics. We also run the matching within part of speech, so that a verb/noun imbalance across frequency bands cannot create a spurious metric divergence.

The closest causal work is Slangvolution. It learns a causal graph with the PC-stable algorithm over 200 Twitter words and estimates the effect of a binary slang label on drift, reporting an average causal effect of 0.084. Its outcome, average pairwise distance between contextual embeddings, is itself frequency-sensitive, and its treatment is not frequency. Our work differs on three counts: frequency is the treatment, the outcome's frequency dependence is removed by design rather than modeled, and the lexicon spans thousands of words over two centuries.

On the causal-inference side, we reuse caliper nearest-neighbor matching and subclassification from Matching on the Estimated Propensity Score. For the graded frequency analysis we use the multilevel extension in Propensity Score Matching and Subclassification in Observational Studies with Multi-Level Treatments. The Importance of Being Causal motivates the concern about scale: at LinkedIn, matching cut a naive 10% lift to 2.4%. That result comes from user-behavior data, though, and says nothing about how large the correction will be for language.

Research Program

The corpora are the Google Books English n-grams used by both Hamilton et al. and Dubossarsky et al., together with COHA, the Corpus of Historical American English. Both are divided into decades. For a baseline decade $t$ and an outcome decade $t+k$, we define treatment by frequency at $t$. Covariates are measured in decade $t-1$ or at $t$ from equal-sized samples, so that no covariate is affected by the treatment. The covariates are part of speech, sense count estimated by clustering $n$ contextual token embeddings, prototypicality (distance to the category centroid, following A bottom up approach to category mapping and meaning change), the prior frequency trend from $t-1$ to $t$, and morphological irregularity. Function words and other words without comparable counterparts fall outside common support. They are excluded, and we report how much of the lexicon this removes.

Measurement equalization comes in two versions. For contextual embeddings, we draw exactly $n$ tokens per word per period and compute pairwise-distance and cluster-distribution drift. For static SGNS embeddings (word2vec-style skip-gram with negative sampling), we build a thinned training corpus. In it, every target word keeps exactly $n$ randomly chosen occurrences per decade, and its other occurrences are replaced with a placeholder. One training run then gives every target the same estimation budget. Alignment across decades uses Temporal Referencing, which avoids Procrustes rotations whose error depends on frequency. We repeat the whole pipeline for $n$ in a small grid and over multiple thinning seeds. The spread across seeds gives the sampling-noise interval.

The propensity model is gradient-boosted classification with cross-validated calibration, alongside logistic regression. We match 1:1 within a caliper and report standardized mean differences for every covariate before and after matching. Next we compute $\tau(n)$ on the real and shuffled corpora. Finally, polysemy replaces frequency as the treatment, and frequency joins the matched covariates.

Baselines

The naive reference point is the Hamilton et al. regression of log drift on log frequency and polysemy, run on standard untrimmed embeddings. It is the analysis behind the reported 48%–88% variance explained. We rerun it on our exact corpus and decade split so its coefficient is directly comparable.

The strongest existing correction is the Dubossarsky et al. negative-control comparison. Their protocol computes the frequency–drift relation on the real and the shuffled corpus and treats the gap as the genuine effect; the reported figures are 67% versus 56% variance explained. We apply this to the same embeddings with the same hyperparameters and tuning budget. A win requires our estimate to satisfy two conditions. First, it must recover a known injected effect in the simulation described below, where theirs does not. Second, on the shuffled corpus it must be indistinguishable from zero across $n$, whereas their unmatched residual stays frequency-dependent.

The mixed-effects covariate model of Cultural Shift or Linguistic Drift? is the strongest covariate-adjustment competitor. It receives the same covariates and the same thinned outcomes, so any difference between it and our estimator reflects matching and overlap, not inputs.

Two ablations isolate each ingredient. The first uses matching without token thinning, which tests whether balance alone helps. The second uses thinning without matching, which tests whether equalized measurement alone suffices. For detection benchmarks, the reference numbers from the file are SGNS with Temporal Referencing at 51.8% average Spearman correlation and 66.5% accuracy on SemEval-2020 (TemporalTeller). EmbLexChange reaches 70.7% Subtask 1 accuracy with frequency-stable pivot words (EmbLexChange). No per-language reported result was found on Lacuna for the top Subtask 2 system, UiO-UvA.

Why not just…?

Why not just regress drift on frequency with covariates, as Hamilton et al. did? Regression cannot fix an outcome whose noise depends on the treatment. The shuffled-corpus result, 56% variance explained with no real change present, shows exactly this. Regression also extrapolates silently across frequency ranges where no comparable words exist. The real threat is a narrower version of this alternative: regression on thinned outcomes. If that model gives the same $\tau$ as matching, then the contribution is the measurement design, not the propensity score, and we will report it that way. The thinning-without-matching ablation settles the question.

Why not just subtract the shuffled-corpus baseline? Shuffling shows that an artifact exists, but it removes all real change at once and does not balance which kinds of words sit in each frequency band. If high-frequency words are disproportionately verbs, as the global-versus-local findings suggest they may be, the shuffled difference can still be confounded. Our estimator keeps the shuffled correction and adds balance. The simulation test below shows whether the shuffled correction alone is biased.

Why not just equalize token counts and skip matching? This is the most serious alternative, because sampling variance is the analytically proven driver in Dubossarsky et al. It would plausibly remove most of the artifact. What it leaves untouched is confounding by polysemy, prototypicality, and prior trend, which are real linguistic causes of change that correlate with frequency. The thinning-only ablation measures how much estimated protection is attributable to those confounders. If that share is negligible, the conclusion is that measurement noise, not confounding, was the whole problem. That is itself a clarifying result for the field.

Why not just test on SemEval human annotations, which are free of embedding noise? The human-annotated change scores in SemEval-2020 Task 1 are the right external check. However, each language has only a few dozen target words, far too few for matching. We use them for validation, not for estimation.

Proposed Experiments and Decisive Checks

The first decisive check is an injected-change simulation. We take pairs of unrelated words, merge their later-period occurrences under one form, and assign the merge probability independently of frequency. This creates synthetic meaning shifts whose frequency effect is exactly zero, and repeating it at many frequency levels lets us test whether each estimator recovers that zero. We then repeat with a planted negative frequency effect of known size. Our estimator should recover both. Naive regression is expected to show a spurious protective effect. The shuffled-subtraction and thinning-only baselines will reveal how much bias each leaves.

The second check runs the full pipeline on the shuffled Google n-gram corpus. There, $\tau(n)$ must be near zero and flat in $n$. The third check applies the estimator to the SemEval-2020 English, German, Latin, and Swedish target words. Using thinned-embedding drift scores, we test two things: whether frequency residualized by matched pairs still predicts graded human change (Spearman correlation), and whether frequency-equalized drift scores reduce the frequency bias the organizers reported, without falling below the TemporalTeller reference of 51.8% average Spearman correlation.

The substantive result is the sign and size of $\tau(n)$ for global and local drift, by part of speech, on COHA and Google Books. It is paired with the matched polysemy effect. Three outcomes would each be informative. A negative $\tau$ that is stable across $n$ would put the Law of Conformity on a causal footing. A $\tau$ near zero with a surviving polysemy effect would reverse the current reading of Dubossarsky et al. A $\tau$ near zero for both would show that the frequency laws are artifacts of how change is measured.