Long Memory in a Word's Usage Stream as a Bin-Free Signal of Semantic Change

Studies of how word meanings change over history almost always start by cutting the corpus into time buckets, such as decades or years, and comparing a word's representation across buckets. The bucket edges are arbitrary. Small buckets leave too little text to estimate anything, and large ones blur the change itself. We propose to skip the buckets entirely. Line up every occurrence of a word in date order, give each occurrence one number that describes how it is being used, and ask whether that sequence has long memory. Long memory here means the word's usage at one point stays statistically tied to its usage far earlier, beyond what a random sequence would show. The tool for measuring this is detrended fluctuation analysis (DFA). DFA sums a sequence into a running total, removes a local straight-line trend within windows of size $n$, and measures how the leftover wiggle $F(n)$ grows as the window widens. If $F(n) \propto n^{H}$, the exponent $H$ (the Hurst exponent) is 0.5 for memoryless noise, above 0.5 for self-reinforcing persistence, and below 0.5 for mean reversion.

The bet is that a word whose senses are actively competing leaves a persistent trace in its usage stream long before a two-period comparison would flag it. That claim can be tested in a way current methods cannot: predict which SemEval words will change using only the earlier corpus. Because DFA works on scales measured in occurrences rather than calendar bins, the scale at which a word's fluctuation curve bends also gives a data-driven answer to the question of how wide a time bucket should have been.

Core Claim

We claim three things about the occurrence-indexed usage series of a target word, after its within-document clustering is controlled for. First, its scaling exponent $H$ at scales above typical document length is significantly higher for words that undergo lexical semantic change than for frequency-matched stable words. Second, this holds when $H$ is measured on the earlier period alone, so long memory forecasts change. Third, the multi-scale structure of DFA carries information that single-scale statistics such as lag-1 autocorrelation or usage variance do not.

The usage scalar for occurrence $i$ is $u_i = \langle \mathbf{e}_i - \bar{\mathbf{e}}, \mathbf{p}_1 \rangle$. Here $\mathbf{e}_i$ is the occurrence's context vector, $\bar{\mathbf{e}}$ and $\mathbf{p}_1$ are the mean and first principal direction of that word's occurrence cloud, and we repeat the analysis on the top few principal directions and on the frozen-model surprisal of the token. We then fit

$$F(n) = \sqrt{\tfrac{1}{N}\textstyle\sum_{t=1}^{N}\big[Y(t) - Y_n(t)\big]^2} \propto n^{H},$$

where $Y(t)$ is the cumulative sum of the centered $u_i$, $Y_n(t)$ is the local linear fit in windows of $n$ occurrences, and $N$ is the number of occurrences.

Relation to Prior Work

The method comes from A Tale of Two Structures: Do LLMs Capture the Fractal Complexity of Language?. That paper turns token-level log-perplexity into a time series and finds human text sits near $H \approx 0.65$, while greedy decoding and instruction tuning distort this. It studies one document at a time, not one word across history; we carry its series-construction idea into the diachronic setting. The same paper also constrains us. It shows that interventions that smooth a generator inflate $H$, which is why we refuse to measure $H$ on model-smoothed embedding trajectories.

The reason to detrend at all comes from Scaling and memory in the non-Poisson process of limit order cancelation. It shows that regime-driven trends masquerade as memory unless they are removed. Historical corpora are full of such trends, including shifts in genre mix and print quality.

The motivating gap is the unresolved conflict over whether semantic change is smooth or bursty. On the Linearity of Semantic Change finds self-similarity decays linearly with time lag (adjusted $R^2 > 90\%$), using discrete slices. Sequential Modelling of the Evolution of Word Representations finds nonlinear LSTM sequence models beat linear alignment on yearly vectors. Cultural Shift or Linguistic Drift? finds verbs drift regularly while nouns shift in bursts, measured over decades. All three work on a dozen to a few dozen time points, too few to tell persistence from a step from noise. Occurrence-indexed series have thousands of points per frequent word, and the shape of $F(n)$ distinguishes these cases directly.

Continuous diachronic embedding models remove bin edges but replace them with a modeling prior. Dynamic Word Embeddings for Evolving Semantic Discovery penalizes year-to-year movement. Deep Neural Models of Semantic Shift (DiffTime) feeds continuous time through a shared network and reads off velocity. Word2Fun writes each trajectory as a sum of sinusoids. A Bayesian Model of Diachronic Meaning Change (SCAN) smooths sense prevalence with Gaussian Markov random fields. Exploring Word Usage Change with Continuously Evolving Embeddings uses an exponentially weighted running average of contextual vectors.

Every one of these imposes a smoothness or a fixed decay scale on the data, and that is exactly the quantity DFA would try to measure. We therefore treat their trajectories as an ablation target, not an input. We are also the first, to our knowledge, to measure memory without any time-axis model at all.

Research Program

We build occurrence streams for SemEval-2020 Task 1 target words (SemEval-2020 Task 1) wherever document dates within each period can be recovered. For English, that means mapping CCOHA documents back to their COHA years. For German, Swedish, and Latin, we use whatever date metadata survives and report coverage honestly. We do the same for the dated New York Times article corpus used by DW2V and Word2Fun, where dates are known to the day.

Each occurrence gets context features from two sources. One is a frozen contextual encoder. The other is a static bag-of-context-words vector projected through a fixed SVD, which avoids the modern pre-training bias SemEval organizers observed in contextual models. Within a calendar year the true order of documents is often unknown, so we randomize order within each year and treat scales below one year's worth of occurrences as uninformative.

The analysis has three parts. The first is DFA-1 and DFA-2 per word, with $H$ estimated over scales from about ten documents' worth of occurrences up to a quarter of the series. The second is crossover detection, meaning a change of slope in $\log F(n)$. The third is a set of shuffle nulls that each destroy one kind of structure while keeping the others. Shuffling within documents keeps document clustering. Shuffling documents within a year kills within-year topical bursts but keeps year-level drift. A full shuffle gives $H \approx 0.5$ by construction. The quantity we analyze is $\Delta H$, the observed exponent minus the exponent after shuffling documents within years. This isolates memory that spans documents and years from the topical burstiness that would otherwise dominate.

A controlled synthetic study, modeled on DiffTime's blended-word benchmark, establishes what DFA can and cannot see. We splice pseudo-words into the dated NYT stream whose context mixture follows one of four regimes: a sigmoid step, persistent fractional drift with known $H$, white-noise switching, or a mean-reverting oscillation. The test is whether $H$ and the crossover location recover each regime. This study matters because DFA's known weakness is that an abrupt step produces a spurious crossover. We will measure that confusion directly rather than assume it away, and we will add a step-versus-drift discriminant if it is needed.

Baselines

The naive reference point is raw frequency. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change reports that frequency governs the rate of change, so frequency is computed on the same earlier-period data and used directly as a change score. Alongside it we run a usage-dispersion score, the variance of $u_i$ in the earlier period. This is the cheapest stand-in for "this word is polysemous." Beating both requires a significantly higher Spearman correlation with the SemEval graded gold, using a paired bootstrap over target words, and a significant partial correlation for $\Delta H$ after controlling for both.

The key ablation is single-scale memory: lag-1 autocorrelation and a variance ratio at one fixed window, computed on the identical streams with identical shuffle corrections. If these match $\Delta H$, the multi-scale machinery has not earned its place. The criterion for winning is a partial correlation with gold, conditional on the single-scale statistics, that stays above zero across languages.

The strongest existing method on the full two-period task is the UWB system (UWB at SemEval-2020 Task 1). It uses static embeddings aligned with canonical correlation analysis and reports a Subtask 2 Spearman of 0.481 (4th of 32 teams) and a Subtask 1 accuracy of 0.687. EmbLexChange (Unsupervised Embedding-based Detection of Lexical Semantic Changes) reports 70.7% average Subtask 1 accuracy, and Word2Fun reports a Pearson of 0.5355, which is a different metric and not directly comparable.

These systems see both periods, so in the forecasting setting they serve as an upper reference rather than a competitor. For the forecasting task itself, ranking change from the earlier period alone, we found no reported result on Lacuna. In the full-timeline setting, we will rerun UWB's alignment pipeline on our corpus with the same embedding dimension and a matched hyperparameter search. We count as a win either exceeding 0.481 or, more realistically, adding significant variance to UWB's score in a combined ranker.

The second ablation runs DFA on DW2V trajectories (yearly NYT), sweeping the smoothing weight $\tau$. This tests whether $H$ measured on smoothed trajectories tracks $\tau$ more than it tracks the gold labels.

Why not just…?

Why not just run DFA on the continuous trajectories from DiffTime, DW2V, or Word2Fun? These trajectories carry 19 to 27 effective time points, far below the several orders of magnitude of scale that DFA needs. Worse, each model builds in smoothness: DW2V through its penalty on year-to-year movement, DiffTime through its shared time network, and Word2Fun through its sinusoid frequencies. Smoothing inflates $H$ in the same way low-temperature decoding does in A Tale of Two Structures. The $\tau$-sweep ablation will show whether trajectory-based $H$ follows $\tau$ rather than the gold labels. If it does not, this alternative is viable and cheaper, and we will say so.

Why not just use frequency or usage variance, given known statistical laws of change? This is the most serious threat. Polysemous, frequent words could both change more and look more persistent, purely through mixture effects. We answer it with partial correlations rather than by argument. The proposal stands only if $\Delta H$ predicts gold change after conditioning on frequency and dispersion. If it does not, the finding becomes a negative result about long-memory accounts of semantic change, which is still worth reporting.

Why not just use lag-1 autocorrelation or one burstiness statistic instead of multi-scale DFA? A single-scale statistic cannot separate within-document topic clustering from memory that spans years, and it cannot tell a step from persistent drift. Both distinctions matter for the linear-versus-bursty dispute. The synthetic regime study and the single-scale ablation test whether that distinction actually improves ranking on real gold. If it does not, we adopt the simpler statistic.

Why not just compare two periods with alignment, as the SemEval winners do? Alignment needs the later period, so it detects change only after the fact and says nothing about its dynamics. Forecasting from the earlier period alone is the one experiment it cannot run.

Proposed Experiments and Decisive Checks

The decisive experiment is forecasting. We compute $\Delta H$ on SemEval earlier-period streams only and correlate it with the Subtask 2 graded gold in each language, against frequency, dispersion, and single-scale memory, with paired bootstrap confidence intervals. A positive, significant partial correlation in at least English and German, the languages with the best date coverage, supports the core claim. A null result under the within-year shuffle control falsifies it.

The second check is the synthetic regime recovery on the dated NYT stream. It reports how often each injected regime is classified correctly from $H$ and the crossover location, and it quantifies how often a step is misread as persistent drift.

The third check targets the bin-size dilemma on full COHA. For changed words, we test whether the crossover scale, converted to calendar time, predicts which bin width maximizes a standard two-period detector's agreement with gold. If it does, DFA answers the question that discrete partitioning leaves to taste. Finally, we test whether verbs and nouns separate in $H$ and crossover, as Cultural Shift or Linguistic Drift? would predict: drift-like persistence for verbs, step-like crossovers for nouns. That would give a concrete multi-scale reconciliation of the linear-decay and bursty-change findings.