Document Consistency That Does Not Erase Rare Types

When a named entity recognizer reads a whole document instead of one sentence at a time, it can copy confident decisions from easy mentions to hard ones. If "President John F. Kennedy" appears early, later mentions of "Kennedy" should probably get the same label. This "one sense per discourse" heuristic has driven document-level gains for twenty years. We argue it has a blind spot once the label set becomes fine-grained and skewed. When two categories overlap in meaning, such as a geopolitical entity versus a location, a facility versus an organization, or DNA versus a protein in biomedical text, propagating labels across a document tends to overwrite the rarer category with its more frequent sibling. Document context then helps the common types and quietly hurts the rare ones that fine-grained annotation exists to capture.

We propose a document-level conditional random field for this setting. A conditional random field (CRF) is a model that scores whole label configurations jointly rather than one token at a time. In our version, the cross-mention coupling does not reward identical labels. It learns which type pairs are compatible across mentions of the same name. The model is trained with a structured loss that charges more for confusing semantically close types and corrects for class frequency inside the global normalization, not after decoding. The core claim can be tested directly: existing consistency mechanisms convert correct rare-type predictions into wrong frequent-sibling predictions at a measurable rate. A sibling-aware, frequency-corrected joint model keeps the document-level gain on frequent types and removes most of that damage.

Why this gap is real

The evidence for the gap is scattered across the file but consistent. Document-level methods enforce agreement. Majority voting overrides local outliers with the dominant document tag (Nobata et al.). DocL-NER refines uncertain tokens by attending to a memory of draft labels from the rest of the document (Leveraging Document-Level Label Consistency for Named Entity Recognition). The Span Graph Transformer links identical mentions across sentences in a sparse graph (Span Graph Transformer for Document-Level Named Entity Recognition). None of these papers reports per-class effects of propagation. All of them treat "same surface form" as evidence for "same label," which is safest when classes are coarse and balanced.

Two other findings indicate where this breaks. Interpretable Multi-dataset Evaluation for Named Entity Recognition shows that label consistency is the strongest predictor of NER performance. It also shows that document context gives 10.07 F1 on OntoNotes weblogs but nothing on short, uncoordinated text, and that CRFs propagate errors when consistency is low. Enhancing Label Consistency on Document-level Named Entity Recognition attributes about 64% of document-level biomedical errors to inconsistent modifiers. So document-level gains depend on consistency, and consistency is exactly what fine-grained sibling types violate.

The class-imbalance literature supplies the mechanism by which frequency enters. Under skew, likelihood training favors majority pathways. Neural Metaphor Detecting with CNN-LSTM Model abandoned the CRF for a weighted softmax so it could tune recall on rare metaphor tokens. That paper constrains our design: a CRF trained naively under imbalance is a known loser, so frequency correction has to be built into the structured objective.

The semantic-overlap literature supplies the loss idea. Not All Negatives are Equal weights confusable negatives more heavily and gains on 32-class emotion classification. It gives no benefit on binary SST-2, which tells us to expect nothing on coarse CoNLL. Boosting Few-Shot Learning With Adaptive Margin Loss scales margins by class similarity, but it uses static word vectors. Both methods classify isolated inputs. Neither touches structured or document-level prediction.

The closest structural precedent is Neural-PCRF. It models pairwise correlations among ultra-fine entity types with a low-rank compatibility matrix and unrolled mean-field inference. That is the inference machinery we reuse. Neural-PCRF couples the labels of a single mention, though. We couple the labels of different mentions across a document. That changes what the compatibility matrix means: it becomes "if this mention is a GPE, how plausible is it that a later mention of the same name is a LOC?" rather than label co-occurrence within one mention.

Core Claim

Fine-grained document-level NER fails in a specific, measurable way, which we call sibling overwrite. Fixing it requires two ingredients together. The first is cross-mention coupling whose compatibilities are learned per type pair rather than fixed to identity. The second is a training objective that is cost-sensitive to semantic closeness and corrected for class frequency inside the CRF's normalizer. Neither ingredient alone should suffice. Coupling without frequency correction still drifts toward head types. Frequency correction without coupling forgoes the document gain.

Research Program

The model has two layers on top of a shared pretrained encoder that reads the document in overlapping windows. The lower layer is an ordinary linear-chain CRF over BIO tags within each sentence. It is kept because it prevents invalid tag sequences, and boundary errors are a large share of document-level mistakes. The upper layer is a document graph whose nodes are detected mentions. Edges link mentions that share a normalized surface form or an alias, such as a head-word match or an acronym expansion, in the spirit of the alias resolution in Zhou et al.. We build edges with string rules rather than a parser, which avoids the upstream dependence that the file flags for the Span Graph Transformer.

A type assignment $c$ for all mentions in a document $x$ gets the score

$$ S(c \mid x) = \sum_i \psi_i(c_i) + \sum_{(i,j) \in E} g_{ij}\, u_{c_i}^{\top} v_{c_j}, $$

Here $\psi_i$ is the encoder's score for mention $i$ taking type $c_i$. $E$ is the set of document edges. $u, v \in \mathbb{R}^R$ are low-rank type embeddings, as in Neural-PCRF, so $u_a^\top v_b$ is the learned compatibility of types $a$ and $b$ across coreferent-looking mentions. $g_{ij} \in [0,1]$ is a learned gate computed from the two mention contexts. The gate lets the model turn coupling off where documents are uncoordinated, which is the WNUT failure documented by Fu et al. Inference runs a fixed number of mean-field updates unrolled as a recurrent network.

Training uses a softmax-margin objective. The log-partition function is computed over configurations whose scores are augmented by a cost

$$ \Delta(c, c^*) = \sum_i \big( w(c_i, c_i^*) - \tau \log \pi_{c_i} \big), $$

where $c^*$ is the gold assignment, $w(a,b)$ is larger when types $a$ and $b$ are semantically close, $\pi_a$ is the training frequency of type $a$, and $\tau \geq 0$ controls frequency correction. The cost is used only during training, so test-time decoding stays an unbiased maximum of $S$. The sibling cost $w$ is estimated from the confusion matrix of a held-out sentence-level model rather than from static embeddings. This gives the context-dependent notion of overlap that the file notes CRAML lacks. The frequency term pushes the normalizer to spend probability on rare types. This is how we reconcile the CRF with imbalance without retreating to a weighted softmax.

The analysis side is as important as the model. We will compute, for every document-level system, an overwrite rate and a rescue rate. The overwrite rate is the fraction of mentions labeled correctly by the matched sentence-level model and incorrectly after document reasoning. The rescue rate is the reverse. Both are broken down by type-frequency decile and by whether the error lands on a semantic sibling. This makes the hypothesized failure visible independently of whether our model wins.

Baselines

The naive reference point is a sentence-level encoder with a linear-chain CRF, trained with plain likelihood on the same encoder. It anchors the overwrite and rescue analysis and shows how much any document mechanism adds. A second simple reference is the same model followed by document-level majority voting over same-string mentions, as in Nobata et al.. This is the purest form of the consistency assumption we are challenging.

The strongest existing method is the Span Graph Transformer, which reports 91.91 F1 on OntoNotes 5.0. Its recipe also needs a constituency parser and nested-span machinery, so we will additionally reimplement its cross-sentence identical-mention linking on our encoder. That lets us separate its consistency mechanism from its encoder and parsing choices. DocL-NER is the strongest CRF-free document competitor. It reports 92.13 F1 on CoNLL-2003, and we found no reported result on Lacuna for DocL-NER on OntoNotes 5.0, so we will run it ourselves. All reimplemented systems share the encoder, the document windowing, the training data, and an equal hyperparameter search budget of the same number of trials per system, each chosen on the development set.

Beating the field means two things. Our micro F1 must be at least on par with the reimplemented document baselines. Our macro F1 over the 18 OntoNotes types, and our F1 on the bottom-frequency types, must be clearly higher, with confidence intervals from five seeds that do not overlap. The overwrite rate on tail types must be substantially lower. Matching Span Graph Transformer's published 91.91 is a secondary target, because its encoder differs. The ablations are the full model without frequency correction ($\tau=0$), without sibling cost ($w$ uniform), with coupling fixed to identity instead of learned compatibility, and with no document layer but the balanced cost kept. Together these isolate which ingredient removes sibling overwrite.

Why not just…?

Why not just drop the CRF and use a long-context token classifier with frequency-weighted loss? This is the most serious threat. Wu et al. found that a weighted softmax beat a CRF under heavy imbalance, and Fu et al. found that token-level classifiers can beat CRFs on low-consistency data. A long-context encoder might learn implicit consistency, and logit adjustment might protect the tail. We will run this exact system: the same encoder over full-document windows, with a frequency-corrected softmax, and no explicit coupling. If it matches our tail F1 and overwrite rate, we will report that the explicit document CRF is unnecessary and that the contribution reduces to the overwrite diagnosis. Our expectation that it falls short comes from distance. Implicit attention weakens across windows, and frequency weighting alone has no way to say "these two mentions should agree on a coarse category but may differ in a fine one."

Why not just add a label-aware contrastive loss to a document-level tagger? Suresh and Ong show that sharpening sibling boundaries in representation space helps fine-grained classification. Sharper representations make each mention's local decision better, but they do not change what a propagation step does to a rare mention whose name also appears in a head-type context. We will include their loss on top of the reimplemented DocL-NER-style refinement as a baseline. The overwrite-rate breakdown will show whether representation sharpening alone reduces sibling overwrite.

Why not just gate or skip propagation when the model is uncertain? DocL-NER and ConNER already refine only uncertain tokens. The problem is that rare-type mentions are exactly where the local model is least confident, so uncertainty gating sends them preferentially to the refinement step that is biased toward head types. The DocL-NER baseline, combined with our frequency-decile overwrite analysis, tests this directly.

Proposed Experiments

The primary benchmark is OntoNotes 5.0. It is document-structured, and its 18 types contain real sibling overlap (GPE, LOC, and FAC; ORG and NORP; CARDINAL, QUANTITY, and MONEY) with a long tail of rare types such as LAW, LANGUAGE, and PRODUCT. The secondary benchmark is Genia, whose abstracts contain overlapping biomedical types such as DNA versus RNA and cell line versus cell type. Because our model is flat, we evaluate on its outermost-mention flattening, and those numbers are not comparable to the nested 83.36 that Span Graph Transformer reports. CoNLL-2003 is a coarse control. Following the SST-2 null result for label-aware losses, we predict that the sibling cost does nothing there, so a gain on CoNLL would suggest our improvement comes from somewhere other than the stated mechanism. WNUT-2016 is a negative control for coupling: the gate should learn to close, and our model should not fall below the sentence-level reference.

Metrics are micro F1, macro F1, per-type F1 grouped by frequency decile, the overwrite and rescue rates split into sibling and non-sibling errors, and the rate of invalid BIO sequences. The last one checks whether the chain CRF is earning its place.

The decisive check comes first and does not depend on our model. On OntoNotes, majority voting, DocL-NER, and the identity-coupled graph should each show a tail-type overwrite rate that exceeds their tail rescue rate, with the excess concentrated on sibling types. If that pattern does not appear, the premise is wrong and the model is not worth building.

The second decisive check is the ablation grid. Removing either the learned compatibility or the frequency term should restore most of the tail overwrite, while removing both document coupling and coupling learning should lose the head-type gain. The proposal holds only if both ingredients are needed.

The third check is the long-context, frequency-corrected softmax baseline. We treat it as the result that would most change our conclusions, and we will report it as such.