Masked-Context Fingerprints for Growing a Language Model's Vocabulary Over Time

Language on social media changes quickly. New hashtags, slang, product names and deliberately misspelled drug terms appear every month. Old words also pick up new meanings: the 2019 film Us changed what the token "US" refers to, and "speed" can mean a stimulant. Systems that keep a model current by swapping its vocabulary over time, such as Dynamic Vocabulary Replacement, add the new tokens but give them starting vectors that have little to do with what they mean. They also decide what to add purely by counting frequency spikes, so a word whose meaning shifts while its usage rate stays flat is never noticed. We propose a single signal that addresses both problems. To find out what a word means right now, hide it. Mask every occurrence of the word in recent text, ask the current model what belongs in each blank, and average its answers. That average, which we call the word's masked-context fingerprint, serves two purposes. It is the starting vector for a newly added token, and when it moves away from the word's stored embedding, that movement triggers a vocabulary update.

The idea joins two lines of work that so far have not been combined. Entity masking replaces a name with a placeholder so the model must reason from context. Drug-jargon detection already uses it to build a "contextual fingerprint" that holds up when surface forms change (JEDIS), but only as a detector. Nothing it finds is ever written back into the model. Dynamic vocabulary methods do write new tokens back, but they read surface statistics. Using the masked context as the source of new embeddings closes that loop. It is also aimed at the informal, drifting text where the usual starting point, an average of the word's subword pieces, is least trustworthy: "cokaine," "#griezmann," and a re-anchored "US" all break into pieces whose meanings mislead.

Core Claim

For a token $w$ observed in recent contexts $c_1,\dots,c_n$ at period $t$, let $h_i$ be the current model's final hidden state at the position of $w$ after $w$ is replaced by the mask token. Let $g$ be the model's masked-language-modeling head transform, the layer applied just before scoring against the tied embedding matrix. The fingerprint is

$$\mu_t(w) = \frac{1}{n}\sum_{i=1}^{n} g(h_i).$$

Output scores are dot products between $g(h)$ and token embeddings. A new token initialized at $e_w=\mu_t(w)$ is therefore, by construction, the token the current model would most want to predict in exactly the contexts where $w$ appears. This gives a principled starting point that needs no auxiliary model.

For an existing token, we define drift as $d_t(w)=1-\cos(\mu_t(w), e_w)$. A token whose drift is high while its frequency ratio is flat gets a second, sense-specific embedding slot, initialized at $\mu_t(w)$. This is the "silent drift" case that frequency-based trend scores miss by construction (Chen et al.).

The claim has three parts. Fingerprint initialization should reduce the loss spike when new tokens are inserted and shorten recovery. Its advantage over subword averaging should be largest exactly where subwords mislead. And fingerprint-triggered splits should catch meaning shifts that frequency triggers cannot.

Research Program

The base system is DVR from Dynamic Language Models for Continuously Evolving Content. It keeps the embedding table at a fixed size, evicts stale low-frequency tokens, inserts emerging ones (whole hashtags among them), and retrains incrementally on hard examples chosen by embedding shift and live masked-language-modeling loss. That paper supplies our method skeleton and our main baseline. It never masks new entities, and it learns them from raw co-occurrence.

We change only two components. The first is the initializer of inserted tokens. The second is the trigger, which can now add a sense slot for an existing token. Eviction policy, table size and hard-example mining stay as they are, so any gain can be attributed to the fingerprint.

Two pieces of prior work constrain the design. TokAlign aligns vocabularies using GloVe vectors trained on dual-tokenized text. It shows that how embeddings are initialized dominates early stability: initial perplexity falls from 3.4e2 under random initialization to 1.2e2. Embedding structure matters shows that random initialization fails catastrophically and that simple structure-respecting heuristics match an auxiliary-model approach (FOCUS). Both results warn that a new initializer must stay close to the existing embedding geometry. Fingerprints satisfy this automatically because they are produced by the model's own output head.

Both alignment methods, along with VRCP, which averages sub-tokens, were built for moving to a new language. None was tested under informal-domain drift over time. Context-Sensitive Recognition for Emerging and Rare Entities motivates our worry about this setting. On W-NUT 2017, its character-level features reached 58.21% precision on Location but collapsed to 0 F1 on Corporation and Creative Work, categories defined by situation rather than spelling. A surface-derived initializer should fail in the same way.

We also take an open problem from JEDIS. Its masked context alone cannot separate a polysemous word's senses, so it adds a module that looks at the unmasked word. Our split mechanism is the model-internal analogue: one slot for the old sense and one for the new, each with its own fingerprint.

Baselines

The naive reference point is a model that keeps its static subword tokenizer and runs continued masked-language-modeling pretraining on each new period, as fBERT does for offensive text. fBERT reports Macro F1 0.596 on HatEval against 0.483 for vanilla BERT. This baseline gets the same unlabeled text and the same number of steps as every other system. Beating it is necessary but not interesting.

The strongest existing method is DVR with its own initialization and hard-example mining. On 2014 country-hashtag prediction, DVR's token-embedding variant reports Micro-F1 0.628, against 0.467 for a 2014 base model and 0.124 for a stale 2013 model (Dynamic Language Models). We re-run DVR on the same Twitter slices, with the same table size, the same eviction list, the same incremental step budget and the same tuning trials. We count our method as beating it if it reaches higher Micro-F1 at equal steps, or matches 0.628 with clearly fewer steps. If our re-run falls short of 0.628, we will report both figures.

The strongest simple initializer is sub-token averaging as in VRCP: the new token starts at the mean of the embeddings of the subwords it used to break into. We run it inside the identical DVR loop.

The key ablation is the unmasked contextual average. Here the initialization is the mean of $g(h_i)$ computed with $w$ left visible, as its subword pieces. This is the closest competitor to our idea, and it isolates the entity-masking ingredient. If masking matters, the visible version should inherit misleading subword meanings and do worse on exactly the obfuscated and re-anchored tokens.

For the trigger, the reference point is the n-gram trend score from Mitigating Temporal-Drift. That method reports gains of 2.70% to 5.71% F1 over random temporal selection on its 2014–2019 Twitter NER data. We compare its trigger against fingerprint drift at the same budget of vocabulary changes.

Why not just…?

Why not just use sub-token averaging, which is cheap and already works? It works when the pieces of a word share its meaning, as with related languages in VRCP. In our setting that assumption breaks: the pieces of "cokaine," of a hashtag, or of a re-anchored "US" point to the wrong neighbors. The Drexel-CCI collapse on situational entity types is direct evidence that surface-derived signals fail there. This is still a real threat for ordinary neologisms whose pieces are informative, so we stratify results by how far the subword average lies from the token's eventual trained embedding. We will report honestly if the two methods tie on the easy stratum.

Why not just run longer continued pretraining so initialization stops mattering? With enough steps it may stop mattering, and we expect gaps to shrink. TokAlign's recovery within 5,000 steps suggests a good initializer mainly buys speed and stability. The deployment setting, however, is a stream refreshed every period on a fixed budget, which is why DVR emphasizes its 76.9% step saving. We therefore report full learning curves. If fingerprint initialization helps only in the first few thousand steps, the claim narrows to efficiency, and we will state it that way.

Why not just apply mention randomization or entity blinding during downstream fine-tuning? Randomization in the style of Lin et al. keeps a tagger from memorizing names, and on NER alone it may close much of the gap. It does not give the backbone any representation of a new entity or a new sense, so it cannot help tasks that need one, such as hashtag prediction or linking a new entity. We include it as an additional NER baseline and also combine it with our method. If it matches us on NER and we win only on the representation-dependent tasks, that sharpens rather than weakens the claim.

Why not just ask a large language model? JEDIS reports that GPT-4o-mini reaches high recall on drug jargon but low precision, flagging benign language. This option also does nothing to update a deployed encoder. We report it only on the jargon benchmark, for reference.

Proposed Experiments

The first experiment uses DVR's Twitter 2013–2019 stream. Over that period, 78.3% of the top 40k hashtags change. At each yearly insertion we measure the masked-language-modeling loss on inserted tokens at step zero and the steps needed to recover to pre-insertion loss. We then measure country-hashtag Micro-F1 at fixed budgets, against the 0.628 reference.

The second uses the 2014–2019 temporal NER corpus of Chen et al.: train on 2014–2018, test on 2019, with F1 broken down by entity type and by whether the entity appeared in training.

The third uses TempEL, with Accuracy@64 split into continual and new entities. That paper reports a drop of up to 17.9% on new entities when tested out of snapshot. We found no reported absolute Accuracy@64 on Lacuna for its BLINK baseline, so we will re-run BLINK ourselves. TempEL is formal text, so it tests whether the fingerprint transfers beyond social media rather than serving as the core setting.

The fourth uses the JEDIS Reddit and Silk Road benchmarks, 1,000 sentences each. The reference is JEDIS at F1 0.64 and 0.63. We expect gains mainly on polysemous terms such as "speed."

Two checks are decisive. The first is a controlled silent-drift test. In a held-out later period we inject a known meaning change into an existing token while keeping its frequency constant, for instance by redirecting the contexts of a mid-frequency entity onto a common word. We then measure detection precision and recall for fingerprint drift versus the trend score. The trend score should be near chance on this test by construction. If fingerprint drift is not clearly better, the trigger half of the proposal fails.

The second is the masked-versus-unmasked ablation, stratified by subword misleadingness. The masking claim survives only if the masked fingerprint beats the visible-context average specifically on the high-misleadingness stratum. If the two tie everywhere, we drop masking and keep contextual initialization as a simpler result.