Persistent Joint Hypotheses for Training Entity Linkers on Noisy Labels

Entity linking maps an ambiguous name in text, such as "Jordan", to the right entry in a knowledge base, such as Michael Jordan versus the country. The best systems decide all names in a document together, because the right reading of one name constrains the others. Training data for this task is noisy. Wikipedia hyperlinks are incomplete, and labels in new domains often come from weak heuristics. The usual fix for noisy labels is to keep a running record of the model's past predictions for each training example and use that record to spot and correct wrong labels. Doing this jointly, by tracking beliefs about the whole document's interpretation, looks impossible: the number of joint interpretations grows as the product of the candidate counts. Doing it per name throws away the cross-name consistency that makes collective linking work.

We propose to store, for each training document, a small fixed set of $K$ complete candidate assignments ("particles") that persist across epochs. Each time the document is visited, a few cheap local moves update these particles using the current model's scores and knowledge-graph consistency. Their weighted vote supplies cleaned training targets. The idea borrows the particle trick from knowledge-graph query answering, which represents a disjunction as a set of samples instead of expanding it. It applies that trick to the per-example memory that noisy-label methods need. The result is joint, coherence-aware label correction whose memory cost is a few hundred bytes of integer indices per document.

Core Claim

The labeling constraint for a document $d$ with mentions $m=1,\dots,M$ and candidate sets $C_m$ is a conjunction of disjunctions: every mention refers to exactly one of its candidates. Rewriting it as an OR of complete assignments, its disjunctive normal form, gives $\prod_m |C_m|$ terms. This is the same blow-up Query2box runs into when it expands unions into separate boxes. An exact per-document posterior over interpretations would need one number per term. We claim that a persistent set of $K$ sampled terms captures the coherence structure that matters for label correction. The cost is

$$S_{\text{particle}}(d) = K \cdot M \cdot \lceil \log_2 \max_m |C_m| \rceil \ \text{bits},$$

where $K$ is the particle count, $M$ is the mention count, and the last factor is the bits needed per candidate index. The factorized prediction histories used by SELFIE or Temporal Ensembling instead store $q \cdot \sum_m |C_m|$ floats, where $q$ is the history length. With $K=8$, $M=20$ and 64 candidates, a particle state costs 120 bytes per document. A 10-step float history over the same candidates costs about 51 KB. Our central empirical claim is that joint particles correct confusable label noise better than factorized per-mention state of equal or larger size. If that fails, the joint structure is not earning its place.

Relation to Prior Work

Collective linking motivates the proposal. AIDA builds a graph of mentions and candidates and greedily prunes it to one dense, coherent assignment. Graph ranking lets correct entities reinforce each other through random walks. Global ED with BERT resolves mentions one at a time in confidence order and feeds each decision back as context. All three avoid enumerating joint assignments at inference time, and all three train in a memoryless way on whatever labels they are given. We keep their inference style and change what the model learns from.

Strong Heuristics shows that anchoring on unambiguous mentions through Wikidata paths is strong and cheap. We reuse such paths as the coherence term in our particle moves.

The method comes from Query2Particles, which represents a union by merging particle sets instead of converting to normal form. That paper works on knowledge-graph queries, in embedding space, with a fixed $K$ and no training-time memory. We use particles as discrete, persistent, per-example latent state, which it never considers. DUCK adds geometric type constraints to retrieval and is orthogonal: it could serve as the base scorer.

Noisy-label work both motivates and constrains the proposal. SELFIE, Active Bias, ChronoSelect and IW-Fit show that per-example histories help. They also show two hazards. Memory grows with dataset size and history length, and early wrong predictions can lock in as confirmation bias. Our design must answer both hazards, not inherit them. Persistent sampling state per example has precedent in hard-negative mining (Yau et al.), but not as a joint structured posterior.

Research Program

The base model is a global linker in the style of Yamada et al. Candidates come from the BLINK bi-encoder, and a single forward pass scores every candidate of every mention in the document. Each training document keeps $K$ particles, initialized from the observed labels where they exist and from prior popularity elsewhere. When a document enters a batch, the scores the model already computes are reused at no extra GPU cost. A CPU step then performs a few Gibbs-style moves per particle: pick a mention, and resample its candidate in proportion to the exponentiated model score, plus a weighted coherence bonus from Wikidata adjacency with the particle's other assignments, plus a prior toward the observed label.

The training target for each mention is the particle-weighted vote. A mention's label is replaced when the particles agree strongly, which mirrors SELFIE's low-entropy test. Its loss is down-weighted when the particles keep disagreeing, which mirrors ChronoSelect's "boundary" set. A NIL candidate, meaning "no entry in the knowledge base", is included in every candidate set, which addresses AIDA's lack of NIL handling.

We will study three questions. The first is whether persistence matters, that is, whether search effort accumulated across epochs beats fresh search each epoch. The second is whether $K>1$ matters, which bears on whether diverse hypotheses protect against confirmation bias and against AIDA's failure on topically mixed documents. The third is how accuracy and memory scale as training data grows from 1M to 9M Wikipedia hyperlink documents.

Baselines

The naive reference trains the same global linker with plain cross-entropy on the noisy labels. It shows how robust a pretrained encoder already is. Published clean-label numbers bound what is achievable: Global ED with BERT reports 95.0% In-KB accuracy on AIDA-CoNLL after fine-tuning (paper), and AIDA reports 81.91% macro-precision@1 (paper). There is no reported result on Lacuna for entity linking under controlled label noise, so we will rerun every baseline ourselves.

The strongest competitor is factorized stateful correction: SELFIE-style refurbishment and Temporal-Ensembling-style averaging of predictions, applied per mention over candidate distributions. It is the established approach to noisy labels, and SELFIE reports 41.3% test error against 53.2% for standard training on CIFAR-100 with 40% pair noise. We will run it with its history length and threshold tuned on the same validation split and with the same number of tuning trials. It will get one configuration matched to our memory budget and one with unrestricted memory.

We also include a memoryless correction baseline: each epoch, AIDA-style greedy coherent decoding of the training document produces pseudo-labels. This uses knowledge-graph coherence without persistence.

The ablations isolate the key ingredient. The first re-initializes particles every epoch with the same number of moves, removing persistence. The second sets $K=1$, removing hypothesis diversity. The third sets the coherence weight to zero, which reduces the method to per-mention sampling.

All runs share the base model, candidate lists, compute and tuning budget. Beating the strongest baseline means a statistically reliable gain of at least two accuracy points on AIDA-CoNLL test at 40% noise under equal memory, with no loss at 0% noise.

Why not just…?

Why not just use per-mention noisy-label methods like SELFIE? They are the real threat. If confusable noise is mostly fixable from each mention's own score history, they will match us. Our bet is that the hardest errors look plausible locally and are exposed only by the other mentions in the document, as when "Jordan" sits next to "Chicago Bulls". The matched-memory comparison settles this. We will also break results down by how many unambiguous anchors a document contains. Our advantage should appear in documents with few anchors and many ambiguous mentions, and vanish elsewhere. If it does not appear there, the claim fails.

Why not just beam-search or greedily decode each training document every epoch? This gets coherence without stored state. It also re-searches from scratch each epoch, and it inherits AIDA's single-cluster pull on mixed-topic documents. The re-initialized-particle ablation is exactly this alternative at equal compute. If it ties the persistent version, persistence is unnecessary and we will say so. In that case the contribution reduces to coherence-aware pseudo-labeling.

Why not just rely on a strong pretrained linker's robustness and train on noisy labels directly? The naive baseline measures this. A 95.0% clean ceiling suggests the encoder is strong. But SELFIE's gap between 53.2% and 41.3% shows how much uncorrected pair-style noise costs in other domains, and confusable entity swaps are the text analog of pair noise.

Proposed Experiments

The controlled setting injects confusable noise into the AIDA-CoNLL training set at 0%, 20% and 40%, replacing each gold entity with the next most popular candidate for the same name. A second variant drops 50% of labels to mimic how sparsely Wikipedia hyperlinks are annotated. We evaluate In-KB accuracy on the AIDA test set and zero-shot on MSNBC, AQUAINT, ACE2004, WNED-WIKI and WNED-CWEB.

The realistic setting trains on BLINK hyperlink subsets from 1M to 9M documents. There we report accuracy alongside the bytes of training-time state and the CPU time spent on particle moves. We also report precision@1 on the QUOTEBANK "Hard" split against UIScore's reported 0.833 and mGENRE's 0.810 (Strong Heuristics).

Four checks decide the outcome. Joint particles must beat factorized state at equal memory under 40% noise, and the gain must concentrate on low-anchor documents. Persistent particles must beat re-initialized ones at equal moves. Raising $K$ above 1 must reduce the share of corrupted labels that get locked in, measured directly because the injected noise is known. Clean-label accuracy must not drop.