Is It the Name or the Tokens? Separating Memorization from Token Shattering When Models Meet New Entities

Language models often fail on text that mentions people, companies, or places they have never seen. There are two very different reasons this can happen, and the field has been treating them as one. The first is memorization: the model has learned to read a famous name as a lookup key ("Einstein" plus "Princeton" means "employed by") and ignores the sentence around it, so an unfamiliar name removes the crutch. The second is tokenization. Before a model sees text, a fixed tokenizer splits words into pieces from a vocabulary built years earlier. A new or unusual name can be cut into odd fragments, sometimes fragments that look like function words, and those fragments damage how the model reads the whole sentence. We call this "shattering." Both explanations predict that new names break models. They predict different fixes. If memorization is the cause, remove or randomize the name's identity. If shattering is the cause, make the model robust to how names are split.

We propose a controlled study, followed by a falsifiable test, that separates these two causes. We build test inputs in which name familiarity and name shattering vary independently, and we measure how much each contributes to errors in relation extraction, coreference, and named-entity recognition. We then check whether the same two-factor account explains real failures on text that drifts over time. The payoff is practical. The decomposition predicts which family of mitigation should win on future-dated text: causal and information-bottleneck debiasing, or segmentation-aware augmentation. We test that prediction directly.

Core Claim

The literature disagrees on the cause, and no one has measured both factors on the same data. How Fragile is Relation Extraction under Entity Replacements? swaps entities for same-type Wikipedia names and finds LUKE falling from 72.7% to 45.0% F1. It attributes the drop to lost memorization shortcuts, but it never measures how the replacement names tokenize. What's in a Name? reaches the opposite conclusion. It finds that training frequency does not predict brittleness, and that shattering does: "Injuin" becomes [In, ##juin], and BERT misreads it as grammar.

Our claim has two parts. First, these causes can be separated by an experiment in which familiarity and shattering are crossed rather than confounded. Second, entities that emerge over time are disproportionately both unfamiliar and shattered. That combination is the regime where today's debiasing methods are least justified, because each of them assumes a meaningful name representation exists:

Research Program

The factorial probe set. Starting from the test sets of TACRED and ENTRED for relation extraction, and the coreference set used by What's in a Name?, we substitute each entity mention with a same-type name drawn from one cell of a two-by-two-plus design.

The familiarity axis has three levels: famous real names, obscure real names (low web frequency), and invented names. Invented names guarantee no stored facts.

The shattering axis is set by the target model's own tokenizer. We define a shatter index for a name: the number of pieces per character, plus a flag for whether any piece is a high-frequency function-word token. We then select names at the low and high ends. Invented names give clean control on this axis, because we can generate strings that are single in-vocabulary pieces and strings of equal length that split into many fragments. We generate them separately for each tokenizer family (WordPiece for BERT and LUKE, byte-level BPE for RoBERTa and Llama). Famous names that happen to shatter fill the remaining cell.

Each original sentence appears in every cell. Differences are therefore attributable to the name, not the context.

The decomposition. For each model we fit an error model at the mention level:

$$\Pr(\text{error}) = \sigma(\beta_0 + \beta_f\,\text{unfam} + \beta_s\,\text{shatter} + \beta_{fs}\,\text{unfam}\cdot\text{shatter} + \gamma^\top c)$$

Here $\sigma$ is the logistic function, unfam is the familiarity level, shatter is the shatter index, and $c$ is a vector of context controls: sentence length, and whether the original instance is "shortcut-able" in the context-masked test of the ENTRE paper. The coefficients $\beta_f$, $\beta_s$, and $\beta_{fs}$ are the quantities of interest. A large $\beta_{fs}$ would mean that the two causes compound, which is what we expect for genuinely new entities.

Temporal check. The probe set is synthetic, so we test whether its coefficients predict real failures. We use three temporal sources. The first is the Temporal Twitter NER data of Mitigating Temporal-Drift, covering 2014–2019 at 2,000 tweets per year. The second is WNUT2017 (Results of the WNUT2017 Shared Task), whose test entities never appear in training. The third is the year-stratified NYT setup of Towards Robust Named Entity Recognition via Temporal Domain Adaptation and Entity Context Understanding. That paper motivates this work: it shows that a quarter of failures occur when context is sufficient for humans and the surface name is the problem. However, it never looks at segmentation.

On these sources we measure three things:

  • whether the shatter index of entity mentions rises with distance from the training period;
  • whether the coefficients fitted on probes predict per-mention errors on real later-year data better than familiarity alone;
  • whether the interaction term matters.

The mitigation test. The decomposition makes a prediction, and we test it. If shattering carries most of the effect on future-dated text, a training-time intervention that targets segmentation should beat memorization-oriented debiasing there. Our candidate is shatter-matched entity substitution. During fine-tuning, entity mentions are replaced with same-type names, including invented ones, sampled so that their shatter-index distribution matches the distribution observed in recent unlabeled text from the deployment stream. This requires only raw text and a tokenizer, with no labels and no gazetteer. The design differs from ENTRE-style substitution, whose Wikipedia names are mostly well-segmented, and from RESEMBLE, which maps names to well-behaved anchors at test time. If instead familiarity dominates, VIB or the causal method should win and our substitution should not. Either outcome is informative, and we commit to the prediction from the probe-set fit before running the temporal evaluation.

Baselines

Naive reference. Standard fine-tuning with no entity handling, trained on the earliest period (2014 tweets; original TACRED) and evaluated on later periods and on the probe cells. Every method reports its gain over this model under identical encoders: BERT-base, RoBERTa-large, and LUKE for RE; BERTweet for Twitter NER.

Generic typed masking. Replace mentions with type tags such as [PERSON], following the delexicalization tradition the file attributes to Chen and to Levy et al. This removes both memorization and shattering at once, so it is the natural upper bound on "ignore the name." It is known to lose signal: A Variational Approach for Mitigating Entity Bias in Relation Extraction reports masking below VIB on every benchmark. We found no reported masking-only number on Lacuna in the file.

The strongest existing debiasers. Three methods are run under the same encoder, data, and hyperparameter search budget: twenty trials per method, tuned on a held-out split from the training period only, never on future data.

  • VIB. On the OOD splits, the file reports Micro-F1 of 74.8% on REFinD and 58.7% on BioRED, beating the causal method by 1% and 5.3%. This makes VIB the strongest competitor on the memorization side.
  • CORE. It lifts LUKE on the misleading-entity TACRED subset from 51.9% to 69.3% macro-F1.
  • The convex-hull causal method. It reports a 5.7-point OOD gain on EntRED.

RESEMBLE. Inference-time replacement with canonical anchor names. It recovers coreference F1 from 60.91 to 68.31 in What's in a Name?. Because it neutralizes shattering at test time, it is the closest competitor on the tokenization side.

Temporal baselines. For temporal NER we also compare against the trend-based retraining of Mitigating Temporal-Drift, which gained 2.70% to 5.71% F1 over random recent data. On WNUT2017 we use the winning 41.86% entity F1 only as a reference point for difficulty.

Ablations. Two ablations isolate the key ingredient:

  • Matching only. Substitution with an unmatched (Wikipedia-distributed) name pool, so that it differs from our method only in the shatter-matching step.
  • Segmentation only. BPE-dropout applied to entity spans only, which perturbs segmentation without changing identity.

What counts as beating the strongest baseline. Shatter-matched substitution must exceed VIB on later-year Twitter F1 and on the high-shatter probe cells by more than the seed-to-seed spread across five seeds. It must also not fall below VIB on the low-shatter, famous-name cells.

Why not just…?

Why not just scale the model? Tokenization Falling Short finds that Llama3-70B degrades with word length just as Llama3-8B does on token-level tasks. NEO-BENCH finds that GPT-4 fully knows only 53% of recent neologisms. We still include a large instruction-tuned model, prompted, as a probe-set subject. If its $\beta_s$ is near zero, that is a real finding, and it would narrow the claim to fine-tuned encoders.

Why not just mask every entity? Masking removes both causes. The probe set therefore cannot tell us anything about it beyond its accuracy cost, which Should We Rely on Entity Mentions for Relation Extraction? and the VIB paper both document. Under temporal drift, the surrounding context is itself noisier, which should make masking worse still. The temporal evaluation measures this directly. If typed masking matches our method on later-year data, the conclusion is that the name carries no useful signal for new entities, and we would report that.

Why not just apply BPE-dropout everywhere? This is the most serious threat. It is cheap, and it attacks shattering directly. Evaluating Subword Tokenization shows that random splits can create "alien" compositions that hurt out-of-vocabulary generalization, and dropout does nothing about memorized shortcuts. Neither point proves it will lose here, however. We therefore run it both globally and restricted to entity spans, as an ablation with an equal tuning budget. If entity-span dropout matches shatter-matched substitution, we adopt the simpler method and the contribution becomes the decomposition plus that finding.

Why not just retrain on trending recent data? Trend-based selection works, but it needs fresh labels every period. Our substitution uses only unlabeled recent text. We report both alone and combined. The relevant question is whether substitution adds to retraining, not whether it replaces it.

Proposed Experiments

First decisive check: the probe-set decomposition. Using the TACRED/ENTRED and coreference probes, we test whether $\beta_s$ is significantly nonzero after controlling for familiarity, and whether familiarity still matters when invented names are tokenized cleanly. This directly arbitrates between What's in a Name? and the ENTRE account. We expect both coefficients to be nonzero, with their size depending on the tokenizer family.

Second decisive check: drift. On the 2014–2019 Twitter data and the NYT years, we test whether entity shatter index rises with time gap. We also test whether the probe-fitted model predicts per-mention errors in later years. We measure this with held-out log-likelihood and AUC, compared against a familiarity-only model.

Third decisive check: the pre-registered mitigation prediction. On later-year Twitter NER, WNUT2017, and the high-shatter probe cells, we compare shatter-matched substitution against VIB, CORE, the causal convex-hull method, RESEMBLE, typed masking, and entity-span BPE-dropout. The metric is entity-level F1 for NER and macro and micro F1 for RE. We also report each method's memorization ratio on the familiar-name cells, so that gains on new names cannot hide losses on old ones.

What would refute the proposal. A clean result in which $\beta_s$ is negligible and VIB wins on future data would refute the segmentation half of the account. That outcome is equally worth publishing, because it would tell practitioners to stop worrying about tokenizers for entity robustness.