Spelling-Aware Vocabulary Updates for Language Models That Age

Language models trained on social media go stale within months. New names, hashtags and slang appear, and the model's fixed tokenizer chops them into meaningless fragments. On Twitter, swapping stale vocabulary entries for emerging whole words already helps a great deal. Keeping a new hashtag as one token rather than splitting it into subword pieces nearly doubles accuracy on a hashtag-prediction task (Hombaiah et al.). But new words on social media do not arrive in one clean spelling. A trending term shows up as an abbreviation, a misspelling, an elongated variant, and a phonetic rewrite, all in the same week. A vocabulary-update policy that counts each spelling as a separate candidate either wastes its limited slots on duplicates or never admits the rarer variants, which then stay fragmented.

We propose to update a model's vocabulary over time in units of lexemes rather than strings, where a lexeme is a group of surface spellings that mean the same thing. A small character-reading encoder maps any spelling to an embedding, so all variants of a new term share one learned vector. When a lexeme is evicted to free space, the encoder can regenerate its vector if the term comes back, so nothing is permanently lost. The core claim is testable: under a fixed vocabulary budget, grouping noisy variants before admission covers more occurrences of new terms and gives larger downstream gains on future data than string-level replacement. These gains should be concentrated on noisy variants of terms that emerged after the base model was trained.

Where this sits in prior work

Four lines of work each hold one piece. The first is diachronic language models, meaning models that are updated period by period as language changes. TimeLMs continually pre-trains RoBERTa on 4.2M tweets per quarter and reports roughly a 10% degradation for unrefreshed models after one year, but it keeps the original subword tokenizer frozen (Loureiro et al.). Trend-based data selection picks bursty tweets for retraining NER models and also leaves the tokenizer untouched (Chen et al.). Both motivate the problem: they show that weight updates alone leave new terms fragmented.

The second line is Dynamic Vocabulary Replacement (Hombaiah et al.). It is the closest existing work and the method we build on. It periodically swaps low-frequency tokens for emerging whole tokens and retrains incrementally with hard-example mining. It treats each emerging string as a discrete unit, does not model spelling variation, and discards evicted tokens. We change all three properties.

The third line supplies a mechanism: the Dynamic Phrase Encoder (Liu et al.) maps arbitrary text spans into an existing embedding space without retraining the backbone. It works only on the generation side, leaves the input tokenizer static, and has been tested only on formal text such as WikiText-103 and legal translation. We move this kind of encoder to the input side and train it on character spellings so that it is robust to noise.

The fourth line is lexical normalization, the task of rewriting noisy tokens into standard forms. We reuse its signals for grouping variants, not its pipeline. Random walks on a word–context bipartite graph can link noisy forms to canonical ones without labels (Hassan et al.). BERT can also be adapted for normalization (Muller et al.). However, BERTweet found that rewriting tweets with a normalization dictionary ("hard" normalization) hurt downstream classification relative to lighter "soft" preprocessing (Nguyen et al.). That negative result constrains our design. We never rewrite the input text. Variants share an embedding, while the model still sees which spelling was used through a small per-variant residual vector. This keeps stylistic signals such as elongation available to the model.

Core Claim

Fix a budget of $K$ vocabulary slots that can be refreshed each period. Let $c(\cdot)$ count occurrences of new terms in the next period's text. String-level replacement admits the $K$ most frequent new strings. Lexeme-level replacement admits the $K$ highest-scoring variant groups $G_1,\dots,G_K$, each covering $\sum_{s \in G_j} c(s)$ occurrences. Coverage cannot decrease when variants are grouped correctly. The empirical question is whether the extra coverage, together with sharing statistical strength across rare spellings, turns into downstream accuracy. That would hold if grouping errors are rare enough not to merge distinct meanings. A second claim is that regenerating evicted lexemes from the character encoder closes a known weakness of fixed-capacity replacement: tokens that come back into use after being evicted.

Research Program

The system has three components, built on the TimeLMs quarterly setting.

The first is a candidate miner. Each quarter it scores n-grams with the trend score from Chen et al. and adds a second signal for semantic shift without a frequency spike. That signal is the change in a term's average contextual embedding across quarters, in the spirit of the token-embedding-shift signal of Hombaiah et al. It targets the limitation Chen et al. state explicitly: frequency scores miss words whose meaning changes while their frequency stays flat.

The second is a variant grouper. It clusters candidate strings using three sources of evidence: character similarity after collapsing repeated letters, context-graph proximity following Hassan et al., and agreement of the character encoder. We measure grouping precision on the WNUT 2015 normalization data, using its gold noisy–canonical pairs as known variant groups.

The third is a spelling-aware lexeme encoder. It is a small character-level network that produces an input embedding for a variant group. It is trained, with the backbone frozen, to match the contextual representation that the backbone assigns to the fragmented subword span. This is the same idea as Liu et al.'s alignment loss, applied to the input side. Contrastive pairs come from the grouper. Admitted lexemes get a cached embedding from this encoder plus a per-variant residual, and then go through a short continued pre-training pass. When a lexeme is evicted, only its cached vector is deleted; the encoder can always regenerate it.

Domain mismatch enters as a constraint. Continual adaptation to informal streams may erode formal-text ability, the bidirectional style mismatch described by Bak et al.. We therefore report formal-text pseudo-perplexity on held-out Wikipedia-style text alongside the social benchmarks, and treat a drop on formal text as a cost.

Baselines

The naive reference point is the base RoBERTa model adapted to Twitter through 2019 and then frozen, in both weights and tokenizer. This is the starting point TimeLMs uses. On TweetEval, Barbieri et al. report a global score of 61.3 for generic RoBERTa and 65.2 for continued-pretrained RoBERTa (RoB-RT). We report our own reproduction, since our time splits differ from theirs.

The strongest weight-only competitor is TimeLMs-style quarterly continual pre-training with a frozen tokenizer, run on exactly the same quarterly tweet samples and the same number of training steps as our method. Loureiro et al. report competitive TweetEval results and a win over BERTweet on irony, but the file gives no exact numbers (no reported result found on Lacuna for TimeLMs' per-task TweetEval F1). We report BERTweet (Nguyen et al.) as the static large-pretraining reference.

The strongest vocabulary-updating competitor is Dynamic Vocabulary Replacement with incremental training (Hombaiah et al.). We give it the same slot budget $K$, the same candidate scores and the same steps, and it admits strings rather than groups. On its own 2014 Country Hashtag Prediction task it reports Micro-F1 0.628, against 0.467 for a model trained from scratch on 2014 data and 0.124 for the stale 2013 model. Beating it means exceeding its future-period score on held-out quarters at equal $K$ and compute. On the hashtag task, any setting where we can reproduce its numbers must show a gain beyond seed variance.

Ablations isolate the key ingredients. The first gives DVR a simple elongation-collapsing regex. This is the cheapest form of variant grouping, and the ablation tests whether learned grouping adds anything beyond it. The second initializes new tokens by averaging their subword embeddings instead of using the encoder. The third hard-deletes evicted lexemes instead of regenerating them. The fourth uses a frequency-only miner without the semantic-shift signal. All systems get identical tuning budgets: the same small grid over learning rate and $K$, selected on a validation quarter that precedes the test quarters.

Why not just…?

Why not just keep the subword tokenizer and continue pre-training, as TimeLMs does? A byte-level tokenizer never produces an unknown token, but representing a term is not the same as representing it well. Hombaiah et al. report 0.561 versus 0.314 Micro-F1 for whole versus fragmented hashtags. TweetEval shows generic tokenizers need 6.8% more subword splits on hate-speech slang (Barbieri et al.). Still, continued pre-training could learn to compose fragments well enough over many quarters. The stratified evaluation below settles this. If the frozen-tokenizer baseline matches us on post-2019 terms after four quarters, our claim fails.

Why not just run DVR after a regex that collapses elongations and lowercases? This is the most serious threat, and we include it as an ablation rather than dismiss it. A regex handles loooove but not abbreviations such as tmrw, phonetic spellings such as makn sens, or typos. If most of the coverage gain comes from elongation alone, the learned grouper is not justified. We will report coverage gain split by variant type to show exactly where the regex stops working.

Why not just normalize the text first and feed clean text to the model? This is "hard" normalization, and BERTweet found that dictionary-based rewriting hurt tweet classification (Nguyen et al.). Static dictionaries also cannot keep up with new slang. A learned normalizer such as Muller et al.'s 79.28 F1 system was trained on 2015 data, and nobody has measured how it holds up on later data. We run that normalizer in front of the frozen-tokenizer baseline as a further comparison. As a side result, we report how its normalization accuracy degrades over time, which addresses the unresolved soft-versus-hard question.

Why not just use a character-level model? Character models avoid fragmentation entirely (Buscaldi et al.). They give up large pretrained subword representations and need much longer input sequences. Our encoder applies character modelling only to newly admitted terms. We report sequence length and latency to show the cost difference.

Proposed Experiments

The main setting follows TimeLMs: start from a model adapted through 2019, then run quarterly updates with each method and evaluate each model on the following quarter. Evaluation uses masked-language-model pseudo-perplexity, our own time-split TweetEval tasks, and the Temporal Twitter NER data (train 2014–2018, test 2019) from Chen et al.. Chen et al. report relative gains of 2.70% to 5.71% F1 over random sampling; the file gives no absolute F1 (no reported result found on Lacuna for absolute F1 on this dataset). We also use the Country Hashtag Prediction setting of Hombaiah et al. and WNUT 2017 emerging-entity NER (Derczynski et al.), where the top shared-task system scored 41.86 entity F1. WNUT 2017 bans training-set entity names from its test set, which makes it a direct test of unseen-term handling. Grouping quality is checked intrinsically on WNUT 2015 normalization pairs.

The decisive check is stratified. We split test tokens along two axes: whether the term is new since the base period, and whether it appears in a canonical or non-canonical spelling. We then compare error rates across methods within each cell. If lexeme grouping is the real mechanism, our advantage over DVR should be near zero on canonical spellings and on old terms, and largest on non-canonical spellings of new terms. A uniform gain across all cells would point to a confound such as extra parameters, not the mechanism we claim.

Three secondary checks follow. The first is a coverage curve: the fraction of new-term occurrences covered as a function of $K$, for string-level and lexeme-level admission. The second is a recurrence test on terms that were evicted and later reappeared, comparing regenerated embeddings against hard deletion. The third is formal-text pseudo-perplexity, to measure the style-forgetting cost.