A language model trained on social media freezes its vocabulary on the day its tokenizer is built. Its vocabulary is the fixed list of word pieces it can read. After that day, every new word, whether a slang term, a pandemic coinage, or a forum euphemism for a drug, is chopped into meaningless fragments. Current methods for keeping such models fresh retrain the weights every few months on new text but never touch this list. Separately, methods that do add words to a vocabulary have only been tested once, on stable technical corpora such as law or medicine. This proposal joins the two. On a stream of text that changes quarter by quarter, the model periodically adds a small number of new tokens. It picks the strings whose fragmentation is costing it the most right now, as opposed to the strings that are merely frequent. It then gives each new token a starting meaning taken from the model's own guess of what fills that word's slot in context.
The bet is narrow and checkable. Gains should concentrate on words that did not exist, or meant something else, when the base tokenizer was built. They should appear in future-time prediction and in downstream detectors of coded language. General ability should be kept intact. If quarterly retraining with a frozen vocabulary already captures new words through their sub-word pieces, the experiments will show that cleanly. That outcome would be a useful negative result for a design choice that every diachronic model currently makes silently.
Core Claim
Diachronic continual pre-training means repeatedly further-training a model on each new time slice of text. Under this regime, a vocabulary update that is drift-aware and gradient-selected, with tokens initialized from the model's own contextual prediction, should reduce degradation on future-time text. The reduction should show up specifically on post-tokenizer words. It should also improve euphemism detection when the updated encoder is dropped into existing pipelines. It should do this better than three alternatives: a frozen vocabulary, frequency-based tokenizer retraining, and a one-time domain vocabulary expansion.
Where This Sits
Diachronic continual pre-training. TimeLMs further-trains a RoBERTa-base checkpoint each quarter on roughly 4.2M new tweets. It shows that a static 2020 model loses nearly 10% after one year of future data. It also shows that updating weights alone lets the model learn new facts, such as predicting "mask" in late-2020 contexts. That last result is the strongest argument against this proposal: compositional sub-word learning may already be enough. Our contribution is to test exactly where it is not enough, since TimeLMs keeps the original RoBERTa vocabulary throughout.
Social-media pre-training and fragmentation. TweetEval finds that continued pre-training on tweets reaches a global score of 65.2, against 61.3 for generic RoBERTa and 61.0 for a tweet model trained from scratch. It also finds that generic tokenizers need 6.8% more sub-word splits on hate speech. This motivates the proposal: fragmentation is measurable and concentrated in the informal, offensive text where drift happens. The paper also constrains it: training from scratch underperformed continued training, which counts against simply rebuilding a tweet-native tokenizer and model.
Vocabulary expansion. Our method is taken from two papers. Gold Panning in Vocabulary (VEGAD) selects new tokens by tracking gradients on both the input embeddings and the output prediction layer. It shows that a small selected subset beats both no expansion and full frequency-based expansion. In its tests, frequency-based expansion dropped GSM8K math accuracy from 22.10% to about 6–9%, while VEGAD kept 15.20%. It leaves initialization unsolved and was only run once on static Chinese legal and medical corpora. exBERT adds an extension vocabulary and extension modules to a frozen BERT for biomedical text, also as a one-time step. We reuse VEGAD's gradient scoring but make it differential across time. We also replace sub-token averaging with a context-derived initialization.
Time conditioning and forgetting. Temporal Attention adds a small time matrix to self-attention and reports Pearson 0.620 for English and Spearman 0.763 for German on SemEval-2020 Task 1. That shared task found static word vectors beat contextual BERT, partly because of the pre-trained model's bias toward modern senses. DAS reduces forgetting across sequential domains through importance-based soft-masking. Both keep a fixed vocabulary. We treat them as complementary components that could be combined with our method, not as competitors to beat.
Downstream coded-language tasks. These supply our evaluation. Self-Supervised Euphemism Detection and FA-Net both run on frozen off-the-shelf encoders, so a better encoder can be swapped in without changing anything else.
Research Program
The method has a selection step and an initialization step, repeated each quarter.
Selection. At the start of quarter $t$, we enumerate candidate strings: whitespace-delimited words, hashtags, and emoji sequences that the current tokenizer splits into two or more pieces. For each candidate $w$ we compute $g_t(w)$, VEGAD's summed gradient norm on the embedding and output rows of $w$'s pieces, over a sample of slice $t$. We score
$$s_t(w) = g_t(w)\,\log\frac{f_t(w)+\epsilon}{f_{<t}(w)+\epsilon},$$
where $f_t(w)$ is $w$'s relative frequency in slice $t$, $f_{<t}(w)$ is its frequency in all earlier slices including the base corpus, and $\epsilon$ is a smoothing constant. The first factor asks whether the model is struggling with this string. The second asks whether the string is new or rising. Static domain jargon that the model already handles scores low on one factor or the other. The top $K$ candidates are added, with $K$ around 500 per quarter and tuned once on the first quarter.
Initialization. For each new token, we mask every occurrence of $w$ in a sample of contexts and run the current model. We take the averaged final-layer representation at the masked position, after RoBERTa's prediction transform. Because input and output embeddings are tied, this vector is the model's own estimate of "the word that belongs here." We use it as the initial embedding, scaled to the norm of existing embeddings. The baseline initialization is the average of $w$'s sub-word piece embeddings.
Training and stopping rule. The model is then further-trained on slice $t$ with the standard masked-word objective under the TimeLMs schedule. We will also test a variant that keeps a small replay of earlier slices. Vocabulary growth and the total parameter count are logged every quarter. If growth stops paying off, which we judge by gains on post-tokenizer words, that is when to stop expanding.
Data. We build a quarterly English tweet stream comparable to TimeLMs: a 2019 base, then quarterly slices from 2020 onward at about 4.2M tweets each. Each slice is deduplicated and timestamped. We add the Reddit and Gab drug and weapon forum corpora (about 1.27M and 310K posts) and the ICED Reddit corpus of about 90M tokens. These are not strongly time-ordered, so they serve as downstream test beds rather than as stream slices.
Baselines
Naive reference: the frozen 2019 checkpoint. Generic RoBERTa further-trained on the 2019 tweet base and then never updated. It fixes the floor and reproduces the spoilage curve. TimeLMs reports nearly 10% degradation after one year, so our reproduction should show a decline of similar size before any comparison is meaningful.
Strongest existing competitor: TimeLMs-style quarterly continual pre-training with a frozen vocabulary (TimeLMs). It is the established method for exactly this stream and already captures new world knowledge. It runs on identical slices, with the same number of updates per quarter and the same learning-rate schedule. It gets the same hyperparameter search budget, which we fix at eight configurations on quarter one. Our only change is the vocabulary step, plus the negligible cost of computing gradients for selection. The file reports no per-quarter numbers for TimeLMs beyond the spoilage figure. Its static anchor is TweetEval's continued-pre-training score of 65.2 (TweetEval).
Beating this baseline means three things together. First, statistically significant lower word-normalized pseudo-perplexity on post-tokenizer words in the next quarter, averaged over at least four quarters. Second, at least one point higher TweetEval global score on time-shifted test data. Third, no more than half a point of loss on the original 2019-era TweetEval splits.
Frequency competitor. Each quarter we retrain a byte-pair tokenizer on slice $t$ and add its top $K$ new merges. These are the frequency methods that VEGAD found destructive. Matching $K$ isolates the selection criterion.
One-time domain expansion. We run VEGAD once on the pooled tweets through the final quarter, with the same total number of added tokens. This peeks at the future, which makes it an upper bound for static selection. If the streaming method matches it without peeking, that is a positive result.
Ablations isolating the key ingredients. We drop the drift factor, keeping gradient only (streaming VEGAD). We drop the gradient factor, keeping drift-frequency only. We swap context-derived initialization for sub-token averaging. A fourth variant adds DAS soft-masking to the full method, to test whether forgetting controls are additive.
Why not just…?
Why not just keep the vocabulary fixed and let quarterly retraining learn new words from their pieces? This is the real threat. TimeLMs learning "mask" shows that weight updates do absorb new meanings when the string already exists or splits into sensible pieces. Our prediction is narrower: the gap opens for strings that are both new and badly fragmented, such as emoji sequences, hashtag coinages, and deliberate misspellings. The per-word-birth-date analysis described below settles this. If the frozen-vocabulary baseline matches us on that subset, the proposal's central claim is wrong, and we will report it as such.
Why not just retrain a tokenizer on new text, or use a byte-level model that never fragments? VEGAD's result counts against frequency retraining: GSM8K fell from 22.10% to about 6–9% under frequency expansion. Our frequency baseline tests whether the same damage appears in an encoder on a stream. A tokenizer-free model avoids fragmentation, but it would have to be trained from scratch. TweetEval found from-scratch tweet training (61.0) below continued training (65.2) at comparable data. We therefore include a character-aware model trained from scratch under matched compute only as a secondary comparator, and we state openly that the matched-compute comparison favors adapting an existing checkpoint.
Why not just prompt a large general model? ToxiCloakCN reports GPT-4o falling from 0.764 macro-F1 on clean text to 0.677 under homophone cloaking and 0.610 under emoji cloaking (ToxiCloakCN). FA-Net's small encoder beat GPT-3.5 and Llama 2 on drug euphemism identification (0.35 against 0.33 and 0.17 Acc@1) at 10–200 times lower cost (FA-Net). We will still run a few-shot Llama-3 reference on the cloze and euphemism tasks so that the comparison is current.
Why not just prepend a date token? Temporal Attention already outperformed date prepending on semantic change. More importantly, prepending does nothing about fragmentation, which is the problem this proposal targets. We nonetheless include Temporal Attention as an add-on variant to test whether time conditioning and vocabulary growth are complementary.
Proposed Experiments
Experiment 1: future-quarter prediction, broken down by word age. This is the primary test. We train on quarters up to $t$ and evaluate on quarter $t+1$. The metric is pseudo-log-likelihood summed over each word's pieces and normalized per word, because per-token perplexity is not comparable across different vocabularies, and ignoring this would quietly favor larger vocabularies. Results are bucketed by word birth date: words present in the 2019 base, and words first seen in each later quarter. A whole-word cloze test is built on held-out sentences containing post-2019 words, using exact-match top-k.
Experiment 2: downstream tasks. We fine-tune each quarter's checkpoint on TweetEval and report the global score, testing both on original splits and on test items re-drawn from later quarters where available. We also swap each encoder into the self-supervised euphemism pipeline and into FA-Net on the drug and weapon datasets, with every other component unchanged. Reference points are FA-Net's 0.35 Acc@1 on drugs and the self-supervised method's 40–45% top-20 detection. We further run CAMIT-style detection on ICED, whose reported top-20 precision is 4.61‰ with recall 0.53 (Impromptu Cybercrime Euphemism Detection). That run tests whether sparse, newly coined euphemisms benefit when they become first-class tokens.
Experiment 3: retention. Retention is measured with 2019-era TweetEval and word-normalized likelihood on the 2019 held-out set, tracked across all quarters. The aim is to detect any VEGAD-style collapse.
Decisive checks. The proposal succeeds only if all three of the following hold. The full method beats TimeLMs-style training on post-tokenizer words in Experiment 1, while the two are tied on base-vocabulary words. This pattern rules out the explanation that the gain is just extra capacity. The drift-factor ablation loses part of that gain. And context-derived initialization beats sub-token averaging on early-quarter cloze, before training has had time to wash out the starting point. If the gains are instead spread evenly across word ages, the mechanism is more parameters, not drift tracking, and we will say so.
References
- TimeLMs: Diachronic Language Models from Twitter
- TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
- Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs
- exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources
- Temporal Attention for Language Models
- SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection
- Continual Pre-training of Language Models
- Self-Supervised Euphemism Detection and Identification for Content Moderation
- Uncovering and Mitigating the Hidden Chasm: A Study on the Text-Text Domain Gap in Euphemism Identification
- ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbations
- Impromptu Cybercrime Euphemism Detection