Building a knowledge base "cold start" means reading a pile of documents and producing a database of people, organizations, and the facts linking them, with no existing catalog to link to. Every time the system meets a name, it must decide whether this is someone already in the database it is building or someone new. Large language models look like natural tools for this decision. In practice, the hard part is not getting them to emit well-formed records. It is stopping them from merging two different people because a similar-looking profile happened to be shown to them. We think this over-merge is the same failure that question-answering readers show when given irrelevant documents: they draw a confident connection to whatever is in front of them instead of saying "none of these." In a knowledge base the damage compounds, because one wrong merge pollutes every fact attached to either person.
We propose a decoder for language-model-driven cold-start construction that makes "new entity" a first-class, calibrated option. The model first reasons in free text, comparing a mention against a small set of retrieved candidate profiles. It then commits under a grammar, a token-level mask that permits only legal outputs, whose entity slot allows exactly the retrieved candidate identifiers or NEW, and whose relation slot allows the current relation inventory or an explicit NEW-REL production that adds a relation to the grammar. The commit decision is reweighted by a contrastive score: how much better the candidate profile explains the mention than an empty profile does. The main empirical contribution is a controlled stress test. We inject confuser entities (same or similar names, different people) into the candidate set and measure how over-merging, and the hop-1 precision that depends on it, degrades as confusers grow, with and without each ingredient.
Core Claim
In language-model cold-start construction, the dominant error under realistic registries is distraction-induced over-merging, not syntactic invalidity. Two fixes should together recover most of the lost multi-hop precision without the cost of global graph inference. The first is an explicit abstain-to-NEW option in the commit grammar. The second is a contrastive, query-likelihood-style commit score. A third ingredient, a relation grammar that grows only through explicit NEW-REL productions, should keep the induced schema compact without requiring a predefined ontology. Each piece is falsifiable on its own, and the ablations are designed so that any piece that does not earn its place will show it.
Relation to Prior Work
Three papers supply the method. Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning masks tokens with input-dependent grammars and restricts entity slots to catalog candidates plausible for the sentence. It reaches 36.0 F1 on SynthIE-text closed extraction and lifts entity disambiguation accuracy from 54.1% to 80.3%. Its candidates are Wikidata names the model already knows from pretraining. In cold start, candidates are strings the system minted minutes ago, and two of them may share a name, so they must be described in context. That is exactly where distraction enters. We reuse its input-dependent masking but add a NEW production and a grammar that changes as the knowledge base grows. Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering shows readers degrade as retrieved documents increase. It recovers accuracy with an "unanswerable" escape instruction and with reweighting by the likelihood of the query given the document, reporting a 231% gain on SQuAD with 100 DPR documents. We transfer both devices from single-answer QA to the merge-or-create decision, which it does not study. Grammar-Aligned Decoding shows token-level masking biases the model's distribution and corrects this with costly resampling of up to 2,000 samples. We avoid that cost by placing the constraint only on a short commit span after free reasoning, where the legal set is a handful of identifiers.
Other work motivates the problem. Probabilistic Inference for Cold Start Knowledge Base Population with Prior World Knowledge treats construction as joint inference with popularity and cardinality priors, using operations that split over-merged entities and prune relations. It reports a 131% relative gain in hop-1 precision. That gain is direct evidence that over-merging and over-attachment are where multi-hop precision is lost. It repairs merges after the fact over classifier outputs. We try to prevent them at commit time, and we test whether its repair still adds value on top. HLTCOE Participation in TAC KBP 2015: Cold Start and TEDL documents power-law noise in slot values and the unsuitability of Open IE output for strict population. It is the classic pipeline we must beat.
Two papers constrain the design. Knowledge Graph Efficient Construction: Embedding Chain-of-Thought into LLMs shows staged reasoning plus a given ontology lifts TekGen Military F1 from 0.2680 to 0.6205 and collapses Movie relation strings from 1,472 to 246. It assumes the ontology, which cold start does not have, and its authors name coreference as the failure. Its numbers tell us the relation inventory must be controlled somehow. Dialogue Ontology Relation Extraction via Constrained Chain-of-Thought Decoding is the closest combination of constraints and reasoning, but it reaches only 9.2–13.7 F1. Its "reasoning" is branch sampling over a masked vocabulary, not a free scratchpad. We read that as a caution against masking during reasoning. Finally, UOREX shows soft clustering makes open relation extractors' confidence track accuracy, reporting B3 F1 of 0.980 at a 90% confidence threshold on TACRED. It is the reference for whether our NEW-REL decisions are calibrated.
Research Program
The system processes documents in a stream against a registry that starts empty. For each mention, a cheap retriever pulls the top $k$ registry entries by name and context similarity. Each entry has a short profile: aliases, type, and up to a few attached facts. The model sees the mention's local document context and these $k$ profiles. It writes free-text reasoning, and then a commit span is decoded under a grammar. The entity non-terminal expands only to the $k$ retrieved identifiers or NEW. The relation non-terminal expands to the current relation inventory or to NEW-REL followed by a short gloss. On NEW, the registry gains an entry. On NEW-REL, the gloss is added as a terminal for all later documents, so the grammar grows with the knowledge base. Relation names therefore cannot proliferate silently; every new one is an explicit, countable decision.
The commit decision is not taken from the masked logits alone. For candidate $c$ with profile $p_c$ and mention context $x$, we compute
$$s(c) = \log P_\theta(x \mid p_c) - \log P_\theta(x \mid p_\varnothing),$$
where $P_\theta$ is the same model scoring the mention context, and $p_\varnothing$ is a profile carrying only the mention's surface name. The term rewards candidates whose attached facts make the context more predictable than the name alone would. A same-name confuser with unrelated facts therefore scores near zero. The final choice combines the grammar-masked commit probability with $s(c)$, and NEW wins unless the best combined score clears a threshold $\tau$, tuned on a held-out year. This adapts the query-likelihood reweighting of the distraction-aware reader to the merge decision.
The analysis centers on a confuser stress test. Starting from TAC Cold Start corpora, we inject confuser entries into the retrieved set at controlled rates. These are drawn from other documents and share the mention's name or a near-variant while referring to a different entity. We also vary $k$ from 1 to 32. This turns the contextual-noise question into a measurable curve of over-merge rate against number of confusers, for each system variant.
Baselines
The naive reference point is per-document unconstrained extraction with JSON parsing, merging entities by exact normalized string match. It shows how much any merging intelligence buys at all.
The strongest classic competitor is the pipeline lineage of KELVIN with joint-inference repair from Probabilistic Inference for Cold Start Knowledge Base Population with Prior World Knowledge. That paper reports relative gains of 19.4% in all-hop F1 and 131% in hop-1 precision over its own baseline, and says it outperformed the top TAC Cold Start 2015 entry. No absolute all-hop F1 value was found on Lacuna, so we will recompute the score of whatever implementation we run rather than quote one. Because the code is not in the file, we reimplement split and prune as a post-processing step. We apply it both to a clustering-plus-LLM-extraction pipeline and to our own output, on the same TAC 2015 queries and assessments. Beating this baseline means a statistically significant all-hop F1 gain over clustering-plus-repair. It also means our system without repair is at least matched by the repair applied to us, which shows prevention rather than a different error profile.
The strongest language-model competitor is two-pass "reason then format." The same model writes free reasoning, then emits JSON with re-prompting on parse failure. It is shown the same $k$ profiles and an explicit instruction to answer NEW if none match. It gets the same model, same retriever, same $k$, and the same prompt-tuning budget, with its threshold-free behavior also given a tuned confidence cutoff for fairness. A second language-model baseline is grammar-constrained commitment without free reasoning, in the style of the input-dependent grammar of Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning. A third is a label-first schema that forces the identifier before the justification, the configuration the reasoning-tax literature predicts to fail.
The ablations remove one ingredient at a time: the NEW production (forced choice among candidates), the contrastive score, the reasoning span, and relation-grammar growth (frozen inventory versus free strings). The ablation removing the grammar but keeping the NEW instruction is the one that tests whether the grammar matters for entities at all. We count the proposal as beating the two-pass baseline only if over-merge rate at 16 confusers drops by at least a third with recall within two points. Those are our pre-registered targets, not figures from prior work.
Why not just…?
Why not just tell the model "say NEW if none match" and parse its free output? This is the real threat. The entity-side grammar may add little beyond guaranteeing that the emitted identifier exists. Our prediction is that the instruction alone helps but stays under-used as confusers grow, mirroring the reader study where the escape instruction needed likelihood reweighting to reach its full effect (Improving Zero-shot Reader…). The grammar-off, NEW-instruction ablation settles it. If it matches the full system, we will report the contrastive score and the stress test as the contribution and drop the claim for entity-slot grammar.
Why not just cluster mentions with embeddings and let the language model only extract facts per document? This is the KELVIN-style division of labor, and it avoids putting candidate profiles in context entirely. But clustering by surface and context similarity is exactly what fails on same-name different people and on dissimilar aliases (HLTCOE TAC KBP 2015). It is also why the joint-inference split operation had work to do. The clustering-plus-repair baseline measures this directly, and confusers are the regime where we expect it to lose.
Why not just put the whole registry in a long context? The registry grows into the thousands of entries. The distraction evidence predicts that more candidate profiles make the merge decision worse, not better. The $k$-sweep in the stress test is that experiment: if over-merging stays flat as $k$ grows for the two-pass baseline, our premise is wrong and we will say so.
Why not just apply split-and-prune repair at the end? Repair is cheap and proven. We run it on every system. If repair applied to the two-pass baseline closes the gap to our system, then prevention at commit time is not worth its cost, and that is a publishable negative result.
Proposed Experiments
The primary evaluation is on TAC Cold Start 2015, which is distributed under LDC license, with 2013–2014 used for tuning $\tau$ and prompts. Metrics are hop-0, hop-1, and all-hop precision, recall, and F1, plus cluster-level B3 on entity merges where gold coreference is available. A known risk is that the assessment pools reflect 2015-era submissions, so new correct responses may be unassessed. We will audit a sample of unassessed responses per system by hand and report scores both with unassessed responses counted wrong and with audit-adjusted scores.
The stress test reports over-merge rate, under-merge rate, and hop-1 precision against confuser count and $k$, for every baseline and ablation. The key figure is the slope of these curves, not a single point.
Relation inventory control is evaluated on TekGen without giving the ontology. We report the number of distinct relations induced and F1 after mapping to gold relations. CoT_Ontology's ontology-given results (0.6205 F1 Military; 246 relations in Movie) serve as an upper reference, not a baseline. Calibration of NEW-REL decisions is checked on the TACRED known/unknown split against UOREX's reported B3 F1 of 0.980 at a 90% confidence threshold, treated as a reference for calibration behavior rather than a direct competitor. As a sanity check that the decoder does not hurt the closed setting, we run it on SynthIE-text with NEW disabled, where the reported grammar-constrained result is 36.0 F1.
All systems use the same open-weight instruction models at two scales (roughly 8B and 70B), the same retriever, and matched token budgets. We report the extra tokens our reasoning span and contrastive scoring cost per mention.
The decisive checks are three. The first is whether over-merge rate for the two-pass baseline rises with confusers; if it does not, the problem we target does not exist at scale. The second is whether the contrastive score, rather than the grammar, accounts for most of the reduction; this decides what the paper actually claims. The third is whether post-hoc split-and-prune on the baseline closes the hop-1 precision gap; if it does, prevention is not needed.
References
- Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Grammar-Aligned Decoding
- Probabilistic Inference for Cold Start Knowledge Base Population with Prior World Knowledge
- HLTCOE Participation in TAC KBP 2015: Cold Start and TEDL
- Knowledge Graph Efficient Construction: Embedding Chain-of-Thought into LLMs
- Dialogue Ontology Relation Extraction via Constrained Chain-of-Thought Decoding
- UOREX: Towards Uncertainty-Aware Open Relation Extraction
- KELVIN: a tool for automated knowledge base construction