Entity linking is the task of deciding which real-world thing a name in a text refers to, for example whether "Rome" means the city or a television series. Linking models learn a strong habit from modern Wikipedia: when a name is ambiguous, guess the entity that is most famous today. In a 2024 news article that guess is usually right. In an 1880 music journal it is often wrong. A violinist who was well known in his own time may now share his name with a modern celebrity, and some entities in old texts are missing from modern knowledge bases altogether. We propose to measure how much of a linker's accuracy on historical text comes from this "famous-today" shortcut, and to train against it with a focal-style loss. The loss down-weights a training example according to how well a frozen name-only model (one that sees the mention string and modern popularity, never the context) already predicts the correct entity, rather than according to the linker's own confidence.
The claim has two parts. Diagnostically, the gap between a linker's overall accuracy and its accuracy on mentions where the modern name-only guess is wrong should widen with the age of the document. Methodologically, focusing training on exactly those mentions should close that gap better than standard focal loss, inverse-popularity reweighting, or mention-masking counterfactual training, and it should do so without hurting accuracy on modern text. If counterfactual training or plain popularity reweighting does as well, the proposal fails, and the experiments are designed to show it.
Why this gap exists
Three findings in the literature motivate the proposal. KE-MHISTO builds a benchmark from 19th-century English and Italian music periodicals. It documents a "popularity wall": standard models score F1 above 0.80 on popular entities and below 0.50 on long-tail ones, and LLaMA 3.3 70B reaches only about 0.31–0.32 strict accuracy on its question-answering split. The paper diagnoses the problem but proposes no training objective. The HIPE-2022 overview reports entity-linking F1 between 0.30 and 0.60 across roughly two centuries of multilingual documents, and Wikidata coverage of 57% on classical commentaries, so "not in the knowledge base" (NIL) is a frequent correct answer. Linking Temporal Records shows that name agreement becomes less trustworthy as the time gap grows, because distinct people come to share names over decades. Its agreement-decay idea supplies our intuition that a matching surface string is weaker evidence in older documents. That work is limited to structured records with exact timestamps; we carry the idea into neural text linking as a property of the training signal.
Two papers in the file constrain the design. Efficient Overshadowed Entity Disambiguation by Mitigating Shortcut Learning names the mention-surface shortcut. It proposes counterfactual training (CFT), which adds a loss that predicts the entity with the mention replaced by a mask token. It reports that CFT beats standard focal loss and full entity masking on short-context challenge sets, and it is the closest existing work. It operates entirely on modern Wikipedia and AIDA, and it applies focal loss uniformly on model confidence. The file's analysis explains why that version fails: a model that has learned the shortcut is already confident on shortcut-solvable examples, so the modulation measures the wrong thing. Focusing on Context is NICE prunes candidates with an upstream type tagger and reaches 59.9 F1 on the ShadowLink "Shadow" subset. However, it falls to 80.4 on AIDA against GENRE's 93.3, which warns that hard structural filters cost general accuracy. Finally, the "NegCLIP illusion" in The Role of Linguistic Priors in Measuring Compositional Generalization shows a debiasing method appearing to help while actually widening the gap between overall accuracy and accuracy on hard cases, from 3.43 to 13.24. This is why we report hard-subset accuracy, not just overall F1.
Core Claim
Let $b(e^\ast \mid m)$ be the probability that a frozen name-only model assigns to the gold entity $e^\ast$ given only the mention string $m$. Concretely, this is the anchor-text prior from modern Wikipedia, with NIL given probability zero unless the name has no candidates. The linker, with predicted gold-entity probability $p_t$, is trained with
$$\mathcal{L} = -\,(1 - b(e^\ast \mid m))^{\gamma}\,\log p_t ,$$
where $\gamma \ge 0$ controls how strongly examples that the shortcut already solves are suppressed. This keeps the form of the focal loss of Lin et al., $-(1-p_t)^\gamma \log p_t$. The difference is that the modulating probability comes from a model of the shortcut, so the weight does not decay as the linker learns to exploit the name.
Reweighting by a bias-only model is a known pattern in debiasing elsewhere in NLP, but no paper in our file applies it to entity linking. The new parts are what follows from making the shortcut model explicitly modern. First, it gives a diachronic diagnostic. The diachronic shortcut gap, which adapts Wu et al.'s linguistic gap, is overall accuracy minus accuracy on mentions where $\arg\max b$ is wrong, plotted against document date. Second, it gives a testable era variant. We replace $b$ with the product of the modern prior and one minus a prior estimated from period text, so that weight concentrates where today's fame and yesterday's fame disagree.
Research Program
We first build the name-only shortcut models. The modern one uses Wikipedia anchor statistics. The era one uses anchor statistics from the training portions of HIPE-2022 collections, binned by decade, and from the cross-edition links of Nordisk familjebok produced by Matching and Linking Entries in Historical Swedish Encyclopedias. We then measure the diachronic shortcut gap for existing linkers on HIPE-2022 newspapers stratified by publication decade, and on KE-MHISTO's linking split stratified by QRank popularity tier. This step stands alone: if the gap does not grow with document age, the temporal framing is wrong and the work reduces to long-tail reweighting.
Next we train an encoder linker on the same backbone and data as the CFT paper: Wikipedia pre-training followed by AIDA fine-tuning, plus historical training data where available. We train it under each objective, holding everything else fixed. NIL mentions receive full weight by construction, since the shortcut never predicts NIL for an ambiguous name. This lets us test whether the loss also improves NIL detection, which is where the Separate Head-Tail CNN work suggests difficulty-focused training helps with an overwhelming null class.
A known weakness of focal-style losses is that they amplify mislabeled examples. Our weights do not change during training, so a mislabeled example never gets more weight than under cross-entropy. Its relative influence still grows when easy examples are suppressed, so we report results under small $\gamma$ and audit the highest-weight training examples by hand.
Baselines
The naive reference is the same linker trained with plain cross-entropy. It shows how large the shortcut gap is when nothing is done, and every other method is reported as a difference from it.
Standard focal loss on the linker's own confidence, as in Lin et al., is the direct ablation of our key ingredient. It is identical to our loss except that $b$ is replaced by $p_t$. We tune $\gamma$ over the same grid. Beating it on the hard subset isolates the effect of conditioning on the shortcut rather than on difficulty.
Inverse-popularity reweighting weights each example by a decreasing function of the gold entity's QRank. It is the simplest competitor and the most dangerous one. It differs from our method only in ignoring whether the name is actually ambiguous, so we give it the same tuning budget.
Counterfactual training from Efficient Overshadowed Entity Disambiguation is the strongest existing method. It reports best overshadowed-entity accuracy on 7 of 9 datasets and runs at 3.3 queries per second versus 0.6 for KBED. The file has no per-dataset number for it: there is no reported result found on Lacuna for CFT on ShadowLink, HIPE-2022, or KE-MHISTO. We therefore rerun it with its published mixing weight retuned on our historical development data, using the same backbone, data and compute, and we also run CFT combined with our weighting.
As external reference points we report NICE's 59.9 Shadow F1 and GENRE's 93.3 AIDA F1, both from NICE. We also report the BELA and mGENRE results that KE-MHISTO found strongest on its bottom popularity tier; the file gives no numeric value for those. Beating the field means a statistically significant gain in hard-subset accuracy and a flatter gap-versus-age slope on HIPE-2022 and KE-MHISTO than CFT, with AIDA accuracy within one point of the cross-entropy model.
Why not just…?
Why not just apply standard focal loss? The CFT paper already found that CFT beats it on challenge sets. The mechanism is that the linker becomes confident through the name itself, so confidence-based modulation suppresses exactly the examples whose context the model never learned. Our ablation replaces $b$ with $p_t$ and nothing else, so this experiment tests that explanation directly.
Why not just reweight by inverse popularity? This is a real threat. If most historical errors involve obscure entities, popularity alone may capture most of the effect. The two methods diverge on obscure entities with unique names, which popularity weighting needlessly up-weights, and on popular entities whose names are shared with older entities, which it neglects. We will report the two subsets separately. If inverse-popularity weighting matches us overall and on both subsets, we will say the shortcut model adds nothing.
Why not just use counterfactual training on historical data? CFT removes the name entirely in its auxiliary loss. It cannot tell an era-misleading name from a helpful one, and the CFT paper itself shows that discarding the name hurts when context is sparse, as it often is in OCR-damaged historical text. It may nonetheless match us. If it does, the combined CFT-plus-weighting run tells us whether the two methods are complementary or redundant.
Why not just use a large LLM with retrieval? KE-MHISTO reports LLaMA 3.3 70B at about 0.31–0.32 strict accuracy on historical QA, finds retrieval only partially mitigates the popularity wall, and finds specialized linkers ahead on the rarest tier. HIPE-2022 notes that Wikipedia enrichment cannot help when 43% of entities are absent from it. We include a retrieval-augmented LLM run on the same KE-MHISTO linking split as a reference point.
Proposed Experiments
The evaluation spans HIPE-2022 entity linking (newspapers stratified by decade, plus ajmc for NIL-heavy text), KE-MHISTO's MMHERCL linking split stratified by QRank tier, and ShadowLink and AIDA as checks that modern accuracy does not regress. Metrics are linking F1, hard-subset accuracy, the diachronic shortcut gap with its slope against document decade, and NIL precision and recall.
Three outcomes decide the proposal. The first is whether the cross-entropy linker's shortcut gap rises with document age; this validates the temporal framing. The second is whether shortcut-conditioned weighting beats confidence-based focal loss, inverse-popularity weighting and CFT on hard-subset accuracy without widening the gap on easy cases, which guards against a NegCLIP-style illusion. The third is whether the era-contrast variant improves over the modern-only shortcut on the oldest decades specifically, which would show that what counts as a shortcut depends on when the text was written.
References
- KE-MHISTO: Towards a Multilingual Historical Knowledge Extraction Benchmark for Addressing the Long-Tail Problem
- Extended Overview of HIPE-2022: Named Entity Recognition and Linking in Multilingual Historical Documents
- Linking Temporal Records
- Efficient Overshadowed Entity Disambiguation by Mitigating Shortcut Learning
- Focusing on Context is NICE: Improving Overshadowed Entity Disambiguation
- The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Models
- Focal Loss for Dense Object Detection
- Matching and Linking Entries in Historical Swedish Encyclopedias
- Distant Supervised Relation Extraction with Separate Head-Tail CNN