During a disaster, a response team keeps a registry of known shelters, road closures, supply sites and relief groups. Thousands of social media posts arrive every hour, and each one has to be checked against that registry. A post might refer to an entry already in the registry. It might describe something genuinely new, such as an ad-hoc shelter or an unmapped bridge collapse. Or it might be general chatter. Three facts make this hard. The registry and the posts have no shared format: rows, JSON dispatches and free text all arrive mixed together. Most of the words in a post are emotional or generic ("urgent", "please help", "flooding everywhere"). And the matching models we have today are built to pick the closest existing entry, so they rarely say "none of these." This proposal argues that the last two problems are one problem. A matcher that confidently links a post to the wrong shelter usually does so because both texts share disaster-generic words. That same weakness is why it fails to flag the post as describing something new.
We propose training a compact matcher with three properties. It compares a post jointly against its top retrieved registry candidates plus an explicit "new entity" option. It is taught to keep its decision unchanged when generic chatter is added or removed. And it is taught to flip its decision when a single identifying detail is swapped, such as a street number, a name, or a shelter number. The bet is that this paired invariance-and-sensitivity training does what neither noise filtering nor out-of-registry detection does alone. It makes the "new entity" decision depend on identifying evidence rather than on vocabulary overlap.
Core Claim
Schema-agnostic entity resolution means matching records without first aligning their columns, by treating each record as a flat sequence of its values. Compact models already do this well on curated benchmarks. AnyMatch fine-tunes a 124M-parameter GPT-2 to reach 81.96 average F1 zero-shot across nine matching datasets, at a small fraction of GPT-4's cost. PromptEM recasts matching as a fill-in-the-blank prompt for low-label settings. Both decide only "match or not" for a pair that has already been formed. Both are evaluated on records made almost entirely of entity-bearing tokens. On the harder format pairs in Machamp, where a structured record must be matched to free text, the best models stay below 70% F1. Separately, out-of-knowledge-base detection, which means deciding that a mention refers to a real entity missing from the reference database, has a strong method in BLINKout. BLINKout adds a dedicated NIL token to a cross-encoder and trains it jointly with candidate ranking, improving NIL-discovery F1 by 30–40% over score thresholds. It assumes clean mentions in formal clinical or encyclopedic text and a homogeneous text knowledge base.
The obvious combination is to bolt BLINKout's NIL token onto an AnyMatch-style serialized matcher. We treat that as a baseline, not as the contribution. Our claim is that this combination will still fail on social media input. Work on contextual distraction (Cho et al.) shows that transformer readers forced to attend over irrelevant text make confident false connections instead of abstaining. A NIL head trained on clean data has never learned that shared chatter is not evidence of identity. What is new here is a training signal that defines the NIL boundary by counterfactual edits. The model sees the same post with generic chatter injected, where its output must not move. It also sees the post with one identifier swapped, where its output must move: to NIL, or to the candidate whose identifier now matches. This directly targets the two error types the file highlights: distraction-induced false matches, and the numeric-identifier confusions that PromptEM reports for transformer matchers.
Research Program
The model is a cross-encoder of AnyMatch scale. Its input is the serialized post (or dispatch), the $k$ candidates returned by a cheap blocking step, and a learned NIL slot. All of these are scored in one listwise softmax. Comparing the candidates together, rather than one pair at a time, matters for the open-world decision. "None of these" is a judgment relative to the whole candidate set, and a pairwise matcher has no natural way to express it. This also sidesteps the tension with Superposition Prompting. That method isolates documents in parallel paths to avoid distraction, which is incompatible with matching because records must attend to each other. We keep cross-attention and instead make the attention robust through training.
Let $x$ be a post, $C$ its candidate set, and $p(\cdot \mid x, C)$ the model's distribution over $C$ plus NIL, with gold label $y$. The training objective is
$$ \mathcal{L} = \mathrm{CE}\big(p(\cdot \mid x, C), y\big) + \lambda\,\mathrm{KL}\big(p(\cdot \mid x, C)\,\|\,p(\cdot \mid \tau_{\text{noise}}(x), C)\big) + \mu\,\mathrm{CE}\big(p(\cdot \mid \tau_{\text{id}}(x), C), y'\big). $$
Here $\tau_{\text{noise}}$ inserts or deletes chatter sampled from real crisis posts. $\tau_{\text{id}}$ replaces one identifying value, such as a number, a proper name, or a street, and $y'$ is the label that results: NIL, or the candidate that now carries the swapped value. The weights $\lambda$ and $\mu$ balance the two edit terms. Identifying values are found by a simple heuristic: tokens that are rare within the registry and that appear in some registry field. This requires no schema alignment. A model that matches on chatter is penalized by the second term. A model that ignores fine-grained identifiers is penalized by the third.
Blocking uses High-Value Token-Blocking. HVTB indexes records only on tokens whose TF-IDF score exceeds a dynamic threshold, and it blocks a million records in under 30 seconds. We modify it in one way: inverse document frequency is computed against the registry rather than the stream, so that bursty event words cannot dominate. Blocking is not the research contribution. We report its recall separately so that matcher results are not confounded by missed candidates. This matters because false NIL decisions caused by retrieval misses are a known failure mode in noisy text.
Baselines
The naive reference point is a fuzzy string matcher. It uses normalized edit distance between post tokens and registry values, a fixed threshold for NIL, and the same blocking. This mirrors the dictionary approach of Stanford-UBC at TAC-KBP, which reached 84.4% micro-averaged accuracy on news and web linking. It is the honest "do nothing clever" system. On microblogs, the closest dictionary-style linker in the file, DBpedia Spotlight, scored 0.1459 F1 on the Locke collection according to NEED4Tweet. We would beat this reference point by any margin on NIL F1 and on match F1 under chatter.
The strongest existing matcher is AnyMatch. We fine-tune it on the same training data. It decides NIL by thresholding its maximum match probability over the candidates, with the threshold tuned on validation data. Its reported number is 81.96 average F1 on its nine zero-shot datasets (AnyMatch). We report no published NIL result for it because none exists: no reported result found on Lacuna for open-world or NIL F1 of any schema-agnostic matcher, nor for Machamp under an open-world split.
The strongest existing NIL method is a BLINKout-style joint ranking and NIL head on the same backbone, with the same serialization, data and tuning budget. This baseline is the contribution minus our two edit terms. Its reference point is the reported 30–40% NIL-F1 gain over thresholds on MM-2014AB and NILK-sample (Dong et al.). The file gives that figure only as a relative gain, so we will rerun BLINKout itself on those two datasets to anchor absolute numbers.
Two ablations isolate the key ingredient: one drops the noise-invariance term and one drops the identifier-swap term. A third control applies generic data augmentation, adding chatter to training inputs without the consistency term. This separates "more noisy data" from "an explicit constraint that the output must not change." All systems share the backbone size, the candidate sets, three seeds, and an equal hyperparameter search budget. Beating the field means a statistically reliable gain over the BLINKout-style baseline in NIL F1 on the noisy test splits, with match F1 no worse than AnyMatch on clean data.
Why not just…?
Why not just strip the chatter with an extractor, then match the clean spans? This is the most serious threat, so it is a full baseline. We run a named-entity and keyword extractor trained on tweets, keep only the extracted spans, and feed them to the BLINKout-style matcher. Its weakness is that identifying context in crisis posts often lives outside clean entity spans ("the church past the second bridge"). Social media entity extraction is itself error-prone: NEED4Tweet reaches only 0.5339–0.5546 F1 on microblog collections even with web search. If the extract-then-match pipeline equals our method on the noise-dose curve below, the honest conclusion is that robustness should be engineered upstream, and we will report it that way.
Why not just prompt GPT-4 with an "answer none if unsure" instruction? An escape-hatch instruction is exactly what DAS uses. We include GPT-4 with DAS-style negation prompting on a subsample as an accuracy ceiling. Two pieces of evidence argue against it as the deployed answer. SKR found that asking a model directly whether it knows something was the least reliable way to detect knowledge boundaries. And AnyMatch reports GPT-4 at 3,899 times the per-token cost of a 124M model, which is prohibitive at firehose volume. The comparison tells us how much accuracy the compact model gives up.
Why not just tune a similarity threshold on a dense retriever? BLINKout showed that fixed thresholds fail on near-miss entities and that a learned NIL head beats them by 30–40% F1. That failure should be worse when chatter inflates similarity between posts and records. The AnyMatch-threshold baseline tests this directly on our splits.
Proposed Experiments
No benchmark in the file combines the three conditions we need: heterogeneous formats, an open-world registry, and social noise. We therefore build one from existing resources, using three conversions. First, we take Machamp's Rel-Text and semi-structured-to-text tasks and turn them into open-world tasks. For a held-out fraction of queries we remove the gold record from the reference side, following the pruning protocol behind NILK-sample and MM-pruned in Dong et al.. Second, we inject real crisis chatter at controlled proportions of 0%, 50%, 80% and 90% of tokens. The chatter is drawn from Twitter crisis corpora of the kind surveyed by Lewis et al., and the injected portions are disjoint from the training chatter. Third, we hand-annotate a smaller natural test set: several hundred real posts linked to a constructed shelter-and-resource registry, labeled as match or NIL. This guards against the synthetic noise being easier than the real thing. MM-2014AB and NILK-sample are used to check that our training does not hurt clean-text NIL discovery.
We measure NIL precision, recall and F1; in-registry linking accuracy; blocking recall; and throughput in records per second. The decisive check is the noise-dose curve. The hypothesis predicts that the BLINKout-style baseline's NIL F1 falls steeply as the chatter fraction approaches 90%, while ours stays nearly flat. A flat curve for the baseline would falsify the premise. The second decisive check is a near-miss slice: post–record pairs that differ only in a numeric or name identifier. Removing the identifier-swap term should collapse performance on this slice specifically. If it does not, that term is doing nothing. The third check is the natural test set. Gains that appear only under synthetic injection and vanish on real posts would mean we have learned the injection distribution rather than the task.
References
- ANYMATCH – Efficient Zero-Shot Entity Matching with a Small Language Model
- PromptEM: Prompt-tuning for Low-resource Generalized Entity Matching
- Machamp: A Generalized Entity Matching Benchmark
- Reveal the Unknown: Out-of-Knowledge-Base Mention Discovery with Entity Linking
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- High-Value Token-Blocking: Efficient Blocking Method for Record Linkage
- Stanford-UBC Entity Linking at TAC-KBP, Again
- NEED4Tweet: A Twitterbot for Tweets Named Entity Extraction and Disambiguation
- Self-Knowledge Guided Retrieval Augmentation for Large Language Models
- Tapping into Social Media in Crisis: A Survey