To measure whether a word changed meaning between two eras, the most successful methods average the word's contextual vectors in each era and compare the two averages. A contextual vector is the representation a model like BERT or ELMo produces for one specific use of a word in one sentence. Two eras that are far apart should yield two averages that point in different directions. These vectors live in 768 or 1024 dimensions. In that regime a basic fact of high-dimensional geometry applies: an average of a few hundred scattered unit vectors is systematically longer than the true mean direction it estimates. The excess length is largest when the word is rare and when its uses are spread out. A cosine between two such inflated averages therefore understates their true similarity. That makes rare, context-diverse words look as if they changed, even when they did not.
This note proposes that a measurable part of what the field reports as semantic change is this high-dimensional noise floor. The noise floor can be removed in closed form. It also explains several open puzzles at once. Frequent words appear to resist change and polysemous words appear to drift faster, which are the two "statistical laws of semantic change". Simple averaging behaves differently from pairwise-distance scoring. Static-embedding change detection peaks at an intermediate dimension. The geometric lens comes from the hyperbolic-embedding literature, where high-dimensional representations were shown to collapse onto a thin shell on which curved and flat geometries become indistinguishable. Diachronic natural language processing has never imported that lens, even though its dominant method operates on exactly that shell.
Core Claim
The dominant contextual method averages each era's usage vectors into a "prototype" and scores change as one minus the cosine of the two prototypes. It is called PRT in UiO-UvA at SemEval-2020 Task 1 and in ELMo and BERT in Semantic Change Detection for Russian. Suppose era $t$ contributes $n_t$ unit-normalized usage vectors $x_i$ with true mean $\mu_t$ and sample mean $m_t$. Then a short calculation, expanding $\|m_t\|^2$ into $n_t$ diagonal terms and $n_t(n_t-1)$ cross terms, gives
$$\mathbb{E}\|m_t\|^2 = \|\mu_t\|^2 + \frac{1-\|\mu_t\|^2}{n_t}.$$
In high dimensions, usage vectors spread over the sphere, so $\|\mu_t\|^2$ can be small. The second term is then comparable to the first even for hundreds of usages. The cross-era dot product $m_1\cdot m_2$ is already unbiased because the two eras are independent samples, so all of the bias sits in the denominator of the cosine. Replacing each squared norm with its within-era U-statistic (the average dot product over distinct pairs of usages) gives a bias-corrected prototype similarity:
$$\hat{s} = \frac{m_1\cdot m_2}{\sqrt{q_1 q_2}}, \qquad q_t = \frac{1}{n_t(n_t-1)}\sum_{i\neq j} x_i\cdot x_j.$$
This estimator costs the same as PRT. The claim is threefold:
- Uncorrected PRT change scores contain a component that scales like $(1-\|\mu_t\|^2)/n_t$. That component is driven jointly by low frequency (small $n_t$) and contextual diversity (small $\|\mu_t\|^2$), which are exactly the two predictors in the laws of Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change.
- Removing the component improves agreement with human-graded change on SemEval-2020 Task 1 and RuSemShift.
- Once the component is removed, the choice of geometry for scoring stops mattering in high dimensions: spherical, Euclidean and hyperbolic versions give the same ranking. This is the "high-dimensional convergence" result transplanted to a new domain.
Relation to Prior Work
Two lines of evidence motivate this. Hyperbolic vs Euclidean Embeddings in Few-Shot Learning proved that high-dimensional hyperbolic prototypes migrate to a thin boundary shell. There, geodesic distance reduces to angular distance, and a fixed-radius sphere matches the hyperbolic model (67.92% vs 67.21% on CUB 1-shot at $d=1024$). That work studies image classification and never asks how prototype estimates behave when the number of samples per prototype varies by orders of magnitude. Varying sample counts are the defining feature of diachronic word data.
Challenges for Computational Lexical Semantic Change argued that aspects of the Law of Conformity are artifacts of noisier embeddings for low-frequency words. It gave no mechanism specific to contextual prototypes and no correction. Our contribution is to supply both, and to predict that the Law of Innovation carries the same artifact through the dispersion term.
Three papers constrain the design. The trainable curvature in Low-Dimensional Hyperbolic Knowledge Graph Embeddings flattens toward zero at high dimension. The radial component in Hyperbolic Hierarchy-Aware Knowledge Graph Embedding for Link Prediction becomes negligible next to the angular one. The survey by Peng et al. reports that hyperbolic gains shrink with dimension. Together these make the obvious move at this intersection a poor bet: building a hyperbolic diachronic embedding model at 768 dimensions is unlikely to help. We therefore treat curvature as an ablation to be tested, not the mechanism.
UWB at SemEval-2020 Task 1 supplies the unexplained phenomenon of static-embedding change ranking peaking at $d\in[100,175]$. We test whether the noise floor explains it.
Research Program
The first component is analytic. We extend the bias expression to non-unit vectors and to the anisotropic case, where all contextual vectors share a common offset direction. We also derive the corresponding bias for Average Pairwise Distance (APD), the main alternative to PRT. APD equals one minus $m_1\cdot m_2$ for unit vectors. Its numerator is unbiased, but it is unnormalized, so it confounds within-era dispersion with change regardless of $n_t$. This gives a concrete, testable account of why UiO-UvA found PRT and APD favored on differently shaped gold distributions.
The second component is the estimator above, applied unchanged to ELMo and BERT usage vectors. For static skip-gram embeddings aligned by Procrustes or CCA, there is no per-usage sample. We therefore estimate each word's noise variance by retraining across bootstrap resamples of each era's corpus and subtract it in the same way.
The third component is a dimension and geometry sweep. Contextual vectors are projected by PCA to $d\in\{16,\dots,1024\}$. Skip-gram embeddings are trained at $d\in\{50,\dots,500\}$. At each dimension we score change with three versions of the prototype distance: corrected and uncorrected spherical, and a Poincaré-ball version obtained by the exponential map at the origin.
Baselines
The naive reference point is raw word-frequency change between eras, which tells us how much any score is merely tracking frequency.
The strongest contextual competitor is uncorrected PRT and APD on historically fine-tuned ELMo and BERT, reproduced from UiO-UvA. That system was the top Subtask 2 submission, but no reported result for its exact Spearman value was found on Lacuna. We will rerun it ourselves. On RuSemShift, the reported reference points are ELMo PRT at $\rho=0.557$ on RuSemShift2 and BERT at $\rho=0.490$ on RuSemShift1 (ELMo and BERT for Russian).
The strongest static competitor is UWB's CCA/Procrustes skip-gram system (UWB). It reports an average Subtask 2 Spearman of 0.481 and a Subtask 1 accuracy of 0.687.
Fairness is easy to enforce because the correction adds no parameters. Every comparison uses identical fine-tuned checkpoints, identical sampled usages and identical layer choices, with the layer selected once on a held-out language and frozen. The only free choices are those of the baseline itself.
Beating these baselines means improving average Subtask 2 Spearman across all four languages, with a paired bootstrap over target words showing the gain is not noise. Given only 30–40 words per language, we will report confidence intervals rather than point wins.
The key ablation applies the correction only to word-era pairs below a frequency threshold. If the gain comes from low-$n$ words, that isolates the mechanism. A second ablation compares corrected spherical against corrected hyperbolic scoring at each dimension, to isolate correction from curvature.
Why not just…?
Why not just subsample every word to the same number of usages per era? This removes the $1/n$ dependence on frequency. It does not remove the dependence on dispersion. At a fixed $n$, a context-diverse word still has a larger $(1-\|\mu\|^2)/n$ term, so the polysemy–change correlation would remain inflated. Subsampling also discards most usages of frequent words, which raises variance. We include equal-$n$ subsampling as a baseline. If it matches the corrected estimator on Subtask 2 and on the null test below, then the frequency story suffices and the dispersion claim is wrong.
Why not just use a permutation null, shuffling usages between eras and reporting change relative to the shuffled score? This is the real threat. A permutation null calibrates exactly the noise floor we describe, and it may perform just as well. Our answer is not that it fails. It costs hundreds of re-scorings per word, whereas the closed form is free. More importantly, the closed form says which quantities drive the artifact, and that is what allows the laws of change to be re-tested. We run the permutation null head to head. If its rankings and ours agree to within bootstrap error, we report the estimator as an analytic shortcut to it, and the scientific weight of the note rests on the laws reanalysis.
Why not just model the eras in hyperbolic space, since meaning change involves broadening and narrowing along a taxonomy? The convergence literature predicts the radial signal vanishes at the dimensions contextual models use: Moreira et al. show this for prototypes, and Pan and Wang for angular dominance. Our geometry sweep tests this directly. If hyperbolic scoring beats corrected spherical scoring at $d\geq 256$, the convergence claim fails in this domain, and that would itself be a notable result.
Why not just fine-tune the contextual model more on historical text? Domain fine-tuning was already necessary for UiO-UvA, and we keep it. However, it changes the vectors, not the statistics of averaging a finite sample of them. The bias holds for any set of unit vectors, so it persists after fine-tuning. Measuring the noise-floor share before and after fine-tuning will confirm this.
Proposed Experiments and Decisive Checks
The decisive test uses within-era random splits. We split each era of the SemEval-2020 Task 1 corpora (task paper) and the Russian National Corpus sub-corpora into two random halves, where true change is zero by construction. Uncorrected PRT should report change that correlates with $1/n$ and with dispersion. The corrected score should centre on zero with no such correlation. If uncorrected PRT shows no frequency or dispersion dependence on these null splits, the core claim is false.
Second, we score all four SemEval languages and both RuSemShift transitions with corrected and uncorrected versions against the baselines above, using Subtask 2 Spearman, Subtask 1 accuracy, and RuSemShift $\rho$. Latin is the language where UWB reports notably worse performance and where corpora are smallest. It is where the correction should matter most.
Third, we refit the conformity and innovation regressions over the full vocabulary of each corpus. Gold labels are not needed for this step. We report how much of the explained variance each law retains after correction. The prediction is a substantial drop for both laws, not only for conformity.
Fourth, the dimension sweep tests whether the static optimum near 100–175 dimensions flattens once bootstrap noise is subtracted. It also tests whether the rankings from hyperbolic and corrected spherical scores converge as $d$ grows. If both predictions hold, the note's thesis is supported. High-dimensional diachronic change scores mostly need a statistical correction, and the choice of geometry matters little.
References
- UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection
- ELMo and BERT in Semantic Change Detection for Russian
- Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change
- Hyperbolic vs Euclidean Embeddings in Few-Shot Learning: Two Sides of the Same Coin
- Challenges for Computational Lexical Semantic Change
- Low-Dimensional Hyperbolic Knowledge Graph Embeddings
- Hyperbolic Hierarchy-Aware Knowledge Graph Embedding for Link Prediction
- Hyperbolic Deep Neural Networks: A Survey
- UWB at SemEval-2020 Task 1: Lexical Semantic Change Detection
- SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection