Researchers who study how a field's language changes over time, such as a decade of NLP papers or a century of French colonial writing, now routinely ask a large language model to name the themes a model has found. The model that writes the names is modern and general-purpose, so it tends to read old text with present-day meanings. A term that meant one thing in 1850 or in 2008 can quietly be given the name it carries today. Two tools that should catch this do not. Similarity between dense sentence embeddings stays near 0.99 while a word's meaning changes completely, and frequency-based detectors fire only when a word suddenly becomes more common.
We propose to track thematic change in a specialized corpus using sparse, human-readable features from an encoder that has been further trained on the full time span of that corpus. Each feature's label is written from one time slice and then tested against the actual feature activations of every other slice. How well a label keeps predicting across slices becomes the measure of drift. When a label that fits 2008 stops predicting what the same feature responds to in 2018, the feature's meaning has moved, even if word frequencies have not. The work produces three things: era-faithful labels, a detector for silent meaning change, and a controlled test of whether training the encoder on the domain matters.
Core Claim
The closest existing work is Disentangling language change: sparse autoencoders quantify the semantic evolution of indigeneity in French. It trains a sparse autoencoder on embeddings of one French word, indigène, drawn from a 125-year library corpus. A sparse autoencoder is a network that rewrites each dense embedding as a few active entries out of thousands, and each entry, called a feature, tends to mean one thing. The paper names features by showing top-activating sentences to GPT-4o and checks each name by asking GPT-4o-mini to predict activations on held-out sentences. Its drift measure is the Jensen–Shannon distance between feature-activation distributions, which diverges linearly even while dense cosine stays near 0.99.
That paper motivates our proposal, and it leaves three things open. It studies a single word rather than the themes of a whole corpus. It uses a static modern CamemBERT checkpoint, so historical text is seen through modern representations. It checks labels on pooled held-out data rather than slice by slice, so it cannot tell whether a label is right for a particular era.
We claim the following. First, an encoder continually pre-trained on the whole diachronic domain corpus, shared across time so that features stay comparable, yields sparse features whose labels are more faithful to each slice than labels from a general modern encoder. Second, the cross-slice predictive decay of those labels detects meaning change that frequency and dense-cosine methods miss.
Research Program
The pipeline has four stages.
Encoder. We take a general encoder and continue its pre-training on the full time-spanning corpus, with samples balanced by time slice. The primary corpora are the S2ORC ACL and Computer Science subsets used by DSNTM, which cover 2006–2019 in two-year bins. A replication uses the arXiv 2012–2024 stream from Continual Neural Topic Model. We use one shared encoder rather than one per slice because separate encoders would put their features in unaligned spaces. Per-slice encoders are tested as an ablation.
Sparse features and topics. We train a top-k sparse autoencoder on sentence activations pooled with equal weight across slices. This counters the temporal imbalance that Disentangling language change and historical-corpus work warn about. We then move from single words to corpus-wide themes by grouping features that co-activate at document level into topics. Each topic keeps its member features, so it can be inspected feature by feature.
Labeling. For every feature $f$ and slice $t$, an interpreter model writes a label from contrastive exemplars drawn only from slice $t$: sentences where the feature fires, set against sentences where it stays silent. This is the contrastive prompting that Movva et al. showed produces testable descriptions. We also sample exemplars at moderate activation levels, not just the top 20, to reduce the maximum-activation bias that DeepDecipher points out.
Era checking. A predictor model reads each label and predicts activations on held-out sentences from every slice $t'$. We define the transfer score
$$S_f(t, t') = \mathrm{corr}\big(\hat a_f(x \mid \ell_{f,t}),\ a_f(x)\big), \quad x \sim \text{slice } t',$$
where $\ell_{f,t}$ is the label written from slice $t$, $a_f(x)$ is the true activation of feature $f$ on sentence $x$, and $\hat a_f$ is the predictor's estimate given only the label. A feature whose meaning is stable has a flat transfer matrix. A feature that has drifted shows off-diagonal decay, and we summarize that decay as a drift score:
$$D_f = \mathrm{mean}_{t}\big[S_f(t,t) - S_f(t,t+1)\big].$$
This score deliberately differs from the Jensen–Shannon distance over activation mixtures used by Disentangling language change. Jensen–Shannon distance says how much usage shifted. $D_f$ says whether the shift changes what a human-readable name should say.
The analysis then asks whether labels written by a modern labeler that pass the slice-$t$ check also avoid anachronism. Domain experts judge whether each label invokes concepts that post-date its slice. For the scientific corpora, ACL and computer-science researchers can recognize anachronisms such as a 2008 cluster being called "transformer pretraining."
Baselines
The naive reference point is the standard practice left open by The Dynamic Embedded Topic Model. D-ETM, a dynamic topic model with static skip-gram word embeddings and Gaussian random-walk topics, is run on the same bins, and its per-bin top-word lists are handed to an LLM to name. D-ETM's authors explicitly leave automated labeling as future work. Because its word embeddings are fixed, it cannot represent a single word changing meaning, so this baseline tests whether our method adds anything over "dynamic topic model plus naming."
We also run DSNTM on the S2ORC subsets, since it was built for them and uses the citation graph. On the CS subset it reports predictive perplexity of about 2156, against about 2519 for D-ETM. We reproduce these numbers as a sanity check on our preprocessing. We do not compete on perplexity, because our goal is label faithfulness and drift detection.
The strongest competitor on the actual question is per-slice TopicGPT. TopicGPT uses GPT-4 to propose and refine topics from sampled documents and a lighter model to assign documents to topics with verbatim supporting quotes. We run it independently within each slice, with era exemplars in context and an explicit instruction to describe period-appropriate usage. Topics are matched across slices by label-embedding similarity.
It is the strongest competitor because it uses a frontier model's broad knowledge directly on era text and needs no training. In static settings it reports harmonic-mean purity of 0.73–0.74 on Wikipedia, against 0.59 for LDA. We found no reported result on Lacuna for TopicGPT under temporal drift or planted meaning change.
The second strong competitor is the pipeline of Disentangling language change as published: a modern general encoder with no domain training, a sparse autoencoder, a GPT-4o interpreter, a pooled GPT-4o-mini predictor, and Jensen–Shannon drift.
For drift detection alone, we include two further baselines. One is dense cosine similarity. The other is the emerging-trend score of Mitigating Temporal-Drift, which compares n-gram frequencies in recent and past windows and improved NER F1 by 2.70%–5.71% on temporal Twitter.
Fairness. Every system sees the same bins and documents. Every LLM-based system uses the same interpreter and predictor models, with a matched budget of prompt calls per topic. Each system gets the same number of hyperparameter trials, selected on a held-out slice.
Ablations. Two ablations isolate our key ingredients. The first swaps the domain-continued encoder for the general modern encoder and for per-slice encoders, with everything else fixed. The second replaces per-slice checking with pooled checking.
What counts as winning. On the planted-drift benchmark described below, our method should beat both TopicGPT and the published SAE pipeline on silent-drift detection AUROC, with confidence intervals that do not overlap. It should also produce a lower expert-judged anachronism rate at equal or better within-slice simulation correlation.
Why not just…?
Why not just prompt a frontier LLM with era exemplars, as in per-slice TopicGPT? This is the real threat. A strong model shown 2008 abstracts may well describe them in 2008 terms. Two weaknesses remain. The first is cross-slice matching: topics are induced separately in each slice and linked by label similarity. That link breaks in exactly the case we care about, because a stable name hides changed usage and a changed name hides stable usage. The second is that TopicGPT has no activation-level check of whether a label fits the text of other eras. The planted-drift benchmark settles this directly. If per-slice TopicGPT matches our detection AUROC and anachronism rate, we will report that the encoder and sparse-feature machinery is not needed and narrow the contribution to the era-checking score, which can be applied to TopicGPT's own quote-supported assignments.
Why not just use dense cosine or frequency bursts? Disentangling language change shows dense cosine stays near 0.99 through a documented shift from botanical to colonial-legal usage. The trend score of Mitigating Temporal-Drift is built on frequency differences, so by construction it cannot see meaning change at constant frequency. Our planted swaps hold frequency fixed on purpose. We still report both methods, because on natural bursty drift they may be competitive.
Why not just keep a modern general encoder, as the indigène study did? This is an empirical question, and the encoder ablation answers it. A Comparative Study of PEFT Methods for Python Code Generation finds domain pre-training predicts downstream quality more strongly than the choice of fine-tuning method, which suggests the encoder matters. That finding comes from code generation, not drift analysis, so it motivates the test rather than deciding it. If the general encoder performs as well, that simplifies practice, and we will say so.
Why not just label the top words of a continual topic model such as CoNTM? Continual Neural Topic Model handles streaming data well. Its authors note, however, that it uses a fixed vocabulary and that standard coherence metrics fail to measure shifts over time. Top-word lists from a fixed vocabulary cannot show a single word changing sense. The D-ETM-plus-naming baseline stands in for this whole family.
Proposed Experiments
The decisive experiment is a planted-drift benchmark built on the S2ORC ACL subset. In later bins only, we rewrite the contexts of chosen target terms by moving in sentences where a different term appears, then substituting the target term into them. The target's frequency is held fixed, and a matched set of control terms is left untouched. Frequency, dense cosine, and topic word lists should all be nearly blind to this change.
The main metrics are AUROC for ranking drifted features and topics above controls, and whether each method correctly locates the onset bin. A detection-onset accuracy near the injection bin with AUROC well above the frequency baseline would support the claim. Parity with per-slice TopicGPT would narrow it.
The second experiment uses natural drift with known history. We replicate the indigène trajectory on the BNF corpus, which tests whether corpus-wide topics recover the botanical-to-legal shift. We also check the documented CS topic merge of video tracking and reinforcement learning in 2018–2019 reported by DSNTM. The metric is whether $D_f$ peaks at the documented times.
The third experiment measures label quality. Metrics are within-slice simulation correlation, expert anachronism rate, and expert agreement with each label, following the understandability test used in DILA. DILA's clinical experts correctly identified the theme of sparse features about 68% of the time, against 42% for dense features, which gives a reference for what human-rated gains look like. We also report NPMI and $C_V$ coherence for each topic's words so the results stay comparable with HiCOT, which reports $C_V$ of 0.451 on 20 Newsgroups with 50 topics. We expect no gain there and treat those scores as a guard against losing basic topic quality.
Finally, because continued pre-training risks forgetting, we measure the domain encoder's performance on the sparse-probing tasks used by ATM, where ATM reports 0.7161 accuracy. This separates better drift features from a simply degraded general encoder.
References
- Disentangling language change: sparse autoencoders quantify the semantic evolution of indigeneity in French
- Dynamic Structured Neural Topic Model with Self-Attention Mechanism
- Continual Neural Topic Model
- Sparse Autoencoders for Hypothesis Generation
- DeepDecipher: Accessing and Investigating Neuron Activation in Large Language Models
- The Dynamic Embedded Topic Model.
- TopicGPT: A Prompt-based Topic Modeling Framework
- Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp
- A Comparative Study of PEFT Methods for Python Code Generation
- DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction
- HiCOT: Improving Neural Topic Models via Optimal Transport and Contrastive Learning
- TIME-AWARE FEATURE SELECTION: ADAPTIVE TEMPORAL MASKING FOR STABLE SPARSE AUTOENCODER TRAINING