Calibrated Online Alarms for Word-Meaning Change

A word's meaning shifts in live text streams. "Dorian" becomes a hurricane, and "coach" moves from carriages to sports. Today we can only notice such shifts after the fact. The best methods split a corpus into two eras, pool every occurrence of a word in each era, and measure how much the word's mix of senses differs between them. This note proposes a detector that watches each word occurrence by occurrence. It sorts every new use of the word into one of a small number of usage bins and raises an alarm when the bin frequencies drift away from a reference period. The key property is that the alarm threshold does not depend on the word, its frequency, or the language model. The bins are built so that each holds equal probability under the reference period. A single simulated threshold table therefore gives a guaranteed average time between false alarms for every word in the vocabulary.

The opportunity comes from putting together pieces that currently live apart. Semantic-change research knows that a word should be treated as a distribution over usage types, not an average vector. It measures change with Jensen-Shannon divergence (JSD), a symmetric measure of how different two probability distributions are, but only between fixed eras. Change-detection research knows how to watch a stream with controlled false alarms, but only for low-dimensional sensor data. Continuous-time language research wants to get rid of arbitrary time buckets, but it has no hypothesis test that separates real shifts from sampling noise. We aim to deliver all three at once: a sense-aware distribution, updated per occurrence, with a false-alarm guarantee.

Core Claim

Equal-mass partitions ("quantile trees") have one property that matters here. Under the reference distribution, each incoming point lands in bin $k$ with probability close to $1/K$, whatever the data look like. That property depends only on how the bins' boundaries are placed, not on how the splitting directions were chosen, provided the directions are fixed independently of the reference data used to place the quantiles. We exploit this in three ways. First, we pick splitting directions per word that separate its senses, using one half of a reference window. Second, we place equal-mass quantile boundaries along those directions using the other half. Third, we monitor an exponentially weighted moving average of bin frequencies. For each monitored word, let $x_t$ be the contextual embedding of its $t$-th occurrence, $b(x_t)\in\{1,\dots,K\}$ its bin, and $\lambda$ a forgetting rate. We track

$$Z_{t,k} = (1-\lambda)\,Z_{t-1,k} + \lambda\,\mathbf{1}[b(x_t)=k], \qquad T_t = K\sum_{k=1}^{K}\left(Z_{t,k}-\tfrac{1}{K}\right)^2 ,$$

and alarm when $T_t > h$. Under no change, the distribution of $T_t$ depends only on $K$, $\lambda$ and the reference size $N$. This follows from the quantile-tree argument, so $h$ can be set once by Monte Carlo simulation to hit a target average run length between false alarms, written $ARL_0$.

Each split is a projection onto one direction followed by a quantile cut. This costs $O(Kd)$ per occurrence, instead of the $O(KMd^2)$ kernel and covariance cost that limits existing detectors. It therefore runs directly on 768- or 1024-dimensional embeddings.

Relation to Prior Work

The mechanism comes from the change-detection side. KQT-EWMA and Leveni et al. build a Kernel QuantTree partition and track an EWMA of bin counts. Its distribution-free statistic allows Monte Carlo thresholds with guaranteed $ARL_0$. However, it uses Mahalanobis-type kernels, works well only up to about $d=28$, and was tested only on sensor and tabular streams. We keep its calibration logic and replace the kernel geometry with sample-split, sense-targeted linear directions. Werenski et al. develop optimal-transport rank statistics for change detection, but in retrospective form. Kim and Choi use text only as a prior for detecting changes in an external numeric series, not in the words themselves.

The motivation comes from semantic change. Giulianelli et al., SenseCluster and Card's substitution method all show that distributions over usage types capture sense gain and loss that averaged vectors miss. All three need corpora pooled into two eras. Card's frequency correction, quantile scaling across a batch of words, cannot be applied to a single streaming word. Schlechtweg et al. document the resulting frequency bias, in which rare words look more changed.

Our design addresses that bias differently. False alarms are controlled per occurrence rather than per calendar period. A rare word does not get more false alarms; it simply takes longer in wall-clock time to reach a decision.

On the continuous-time side, Horn removes time bins with an exponentially weighted running average of BERT vectors, but collapses senses into one vector and gives no test. Tsakalidis and Liakata model drift across 14 annual slices, but retrospectively and with static vectors. Continuous formulations such as Word2Fun and DiffTime model smooth trajectories, but provide no false-alarm control. Our contribution is the missing test, placed on a sense distribution.

Research Program

For each monitored word, the first $N$ occurrences in the stream form the reference window, split into halves A and B.

On half A we choose splitting directions. We compare three options. The first is the top principal directions of the word's token embeddings. The second is directions that separate clusters of masked-language-model substitutes, following Card. The third is random directions, which serve as a control. On half B we grow the tree: at each step, peel off a bin of mass $1/K$ by an empirical quantile cut along one chosen direction.

Two sources of dependence threaten the independence assumption behind the guarantee. Occurrences within one document are correlated, so we monitor at most one occurrence per document, or a document-level block average. Monitoring thousands of words at once multiplies false alarms, so we set per-word $ARL_0$ from a vocabulary-level budget. We report both per-word and family-wide false-alarm rates.

Beyond alarms, the detector returns two further outputs. The first is an estimated change time, obtained by backtracking the EWMA. The second is a post-alarm divergence: the JSD between reference bin mass and recent bin mass. This divergence is the graded-change score, and it is comparable across words because every word shares the same uniform reference.

One honest limitation is that the detector flags any change in how a word is used, including topical bursts. It does not claim to separate "meaning" from "topic." Instead, we measure how often each kind of change triggers an alarm.

Baselines

The naive reference point is sliding-window JSD. We cluster each word's embeddings with K-Means, as Giulianelli et al. do, compute JSD between adjacent fixed windows, and threshold with a permutation test. We will calibrate this threshold on the same reference data to the same nominal false-alarm rate. Beating it means lower detection delay at matched empirical $ARL_0$, or empirical $ARL_0$ that stays at nominal where the repeated window tests drift below it.

The strongest online competitor is KQT-EWMA from Notarianni et al., run on embeddings reduced by principal component analysis to 16 and 28 dimensions. That range is where it is reported to work. It will get the same $K$, $\lambda$, reference size and tuning budget. No reported result was found on Lacuna for KQT-EWMA on text streams, and the file gives no exact delay numbers for its UCI or INSECTS runs, only an up-to-2x delay reduction over QT-EWMA. We will therefore re-run it on those benchmarks to check our implementation before comparing on text.

The strongest graded-change method is Card's substitution-based JSD with quantile scaling. It reports a weighted average Spearman correlation of 0.514 across SemEval-2020 Task 1 and GEMS, against 0.452 for the best static-embedding baseline (Card). For GEMS alone, Giulianelli et al. report about 0.28. Card's method sees both full eras offline, so it is an upper-reference rather than a like-for-like competitor. Matching 0.514 within a few points while only seeing the stream online would count as success. We also run Horn's running average with a permutation-calibrated cosine threshold. We run standard Bayesian online change point detection on per-word frequency as the surface-statistics baseline.

The key ablation isolates the sense-targeted directions. We compare them against random directions with identical tree size, and against a single-bin frequency-only detector. If random directions match, the "sense-aware" part of the claim fails.

Why not just…?

Why not just run sliding-window JSD with a permutation test? Each window test is valid on its own, but the stream applies it repeatedly with no sequential guarantee, so false alarms accumulate. Window length also imposes a fixed timescale, which the time-bucketing critique of Yao et al. identifies as missing bursty shifts. Our matched-$ARL_0$ comparison on the synthetic streams will show whether this matters in practice. If windowed JSD with a Bonferroni-style correction matches our delays, we will report that.

Why not just reduce dimension and run KQT-EWMA? This is the real threat, because it shares our calibration logic. The difference is that principal components chosen for overall variance need not separate senses, and a single global reduction discards per-word structure. The head-to-head at equal $K$ and reference size, especially on synthetic sense-narrowing where the variance directions carry little of the change, decides this. If reduced KQT-EWMA matches, our contribution shrinks to the text-specific handling of documents and vocabulary-wide calibration, and we will say so.

Why not just threshold Horn's running prototype? Averaged vectors barely move when one sense disappears from a balanced mix, as Giulianelli et al. argue. The synthetic narrowing condition is designed to show this failure directly, with the mean held nearly constant while the sense mix changes.

Proposed Experiments

First, calibration. We build null streams by shuffling occurrences within the earlier SemEval-2020 corpus $C_1$ for all four languages, and within single decades of the Corpus of Historical American English (COHA). We check that empirical $ARL_0$ matches nominal across words with frequencies spanning two orders of magnitude. A clear frequency dependence here would falsify the core claim.

Second, controlled change, following the synthetic-drift design of Tsakalidis and Liakata. We inject a donor word's usages into a target word's stream at a known time and rate, to create sense gain. We delete one substitute cluster, to create narrowing. We then measure delay against injection rate.

Third, SemEval-2020 Task 1 as a stream. We shuffle $C_1$ then shuffle $C_2$, which places a known change at the boundary. The binary subtask becomes "alarm within $C_2$." The graded subtask becomes Spearman correlation of the post-alarm JSD with gold scores. The comparison against Card's 0.514 is the decisive one. The file gives no binary-subtask accuracy for SenseCluster beyond its tenth-place rank, so that comparison is by re-run.

Finally, we apply the detector to the chronological NYT snippet stream from Horn and to COHA. There we check whether alarms and estimated change times land on the qualitatively documented shifts: Dorian, coach, disk and tenure. We also check that alarms disappear on time-shuffled versions of the same streams.