Tagging every word of a long document is a common need, for example marking which spans are people, drugs, or organizations. Encoders with a fixed input length handle this by cutting the document into chunks, and each chunk is then tagged without seeing the others. A mention like "the company" or "she" in chunk 9 often needs a fact stated only in chunk 2, such as which company, or that she is a ship and not a person. Current fixes either widen the encoder's window, which costs memory that grows quadratically with length, or link a word to its exact repeats elsewhere in the document. Exact-repeat linking misses pronouns, aliases, and descriptions, and those are the cases where cutting the document hurts most.
We propose sending something much smaller across chunk boundaries: the tagger's own predictions. The tagger reads one chunk at a time. For each mention it predicts the usual boundary-and-type tag, and in the same step it predicts which entry of a shared, document-wide entity memory that mention refers to. A second pass re-tags every chunk while reading the memory built from all chunks. The type evidence gathered for "Florence" in chunk 2 can then reach "she" in chunk 9, even though the two strings are different. This is the joint-labeling idea applied to fragmented documents. In joint models, a later decision can overrule an earlier one. Here, the linking decision and the document-level evidence it brings back can correct the local tag.
Core Claim
Under a hard per-chunk context budget, a tagger that jointly predicts boundary, type, and a soft link to a document-level memory of predicted mentions will recover most of the accuracy lost to chunking. The gains should fall mainly on mentions whose disambiguating evidence lies in another chunk under a different surface string. This is the case that exact-string methods cannot reach by design. On documents longer than a long-context encoder's window, we expect the memory approach to match or beat full attention at a fraction of the memory cost.
Where This Sits
The fragmentation literature explains why chunking hurts but almost never works at the token-label level. Late Chunking encodes the whole document before pooling chunk vectors. It raised the similarity of an anaphoric sentence to its referent's query from 0.75 to 0.85, which motivates our focus on anaphora, but it produces retrieval vectors, not tags, and it hits a GPU memory wall beyond 8,192 tokens. EBCAR shows that passage-level embeddings with document-ID-masked attention can do cross-passage reasoning cheaply, reaching 64.92 nDCG@10 on ConTEB at 29.33 queries per second. We borrow its lesson that compact cross-segment representations suffice, and move it from passages to mentions.
Two document-level tagging methods are the closest prior work, and both constrain our design. Doc-ARC attends from each token to distant sentences that contain the identical word. It reached 84.25 F1 on OntoNotes1000, against 82.20 for the sentence-level model, with +17.6 F1 on the rarest entities. That shows distant context matters, but the identity key cannot follow "she" back to "Florence". SGT builds a document graph over candidate spans, linking identical strings and nested spans. It reports 91.91 F1 on OntoNotes 5. Its candidate spans come from a frozen external parser, so parser errors are never revisited. Our model replaces both hand-built keys with a learned link that is predicted jointly with the tag, and it proposes spans itself.
The method comes from joint sequence labeling. We use composite boundary-plus-polarity tags from Bing et al. and Peng et al. to fuse boundary and type. Following the warning about label-space explosion, we keep the link as a separate pointer rather than multiplying it into the tagset. From the S-LSTM joint segmentation-and-labeling model we take exploration training, in which the downstream component is trained on the system's own imperfect upstream predictions rather than on gold ones. That model reduced segmentation error by 30% on average. A truly joint segmentation-and-parsing model supports the general premise that letting the later decision select among earlier hypotheses beats committing early.
Research Program
The base tagger is a standard pretrained encoder applied to chunks of $B$ tokens, with $B$ fixed across all systems. Its output layer predicts a composite tag (B/I/O crossed with entity type) for each token.
For each predicted mention, it also outputs a key vector $k$. The mention is attached to memory slots by softmax attention against the slot keys, with one extra "new entity" option that opens a slot. Each slot $j$ stores a key $k_j$ and a running type distribution $p_j$, the attention-weighted average of the type posteriors of the mentions assigned to it.
In the second pass, token $i$ with representation $h_i$ reads the memory as
$$r_i = \sum_j \operatorname{softmax}_j(h_i^\top W k_j)\,[k_j; p_j],$$
where $W$ is a learned bilinear map. The read vector $r_i$ is concatenated to $h_i$ before re-tagging. Because the memory holds predicted labels and keys rather than raw hidden states, its size grows with the number of entities, not with document length.
Training runs both passes end to end with the tagging loss. The memory in pass two is built from pass-one predictions with exploration, so the re-tagger learns to tolerate wrong links and wrong types.
Link supervision is the one design choice we treat as an open question. In the default setting the link is latent and trained only through the tagging loss. In the variant setting it receives an auxiliary loss from OntoNotes coreference chains. Comparing the two tells us whether explicit linking supervision is necessary or whether the tagging loss alone induces useful links.
Baselines
The naive reference is the base tagger on non-overlapping $B$-token chunks with no cross-chunk information. Every gain is measured against it.
The first strong competitor is the same tagger on sliding windows with 50% overlap and score averaging on shared tokens. This is what practitioners actually deploy, and it removes most artifacts caused by entities cut in half.
The strongest existing document-level method is Doc-ARC. It is the only cited system with a reported long-document tagging number: 84.25 F1 on OntoNotes1000 Doc-ARC. SGT is a second strong method, but its 91.91 F1 is on standard OntoNotes 5, not the long-document subset, and the file reports no OntoNotes1000 number for SGT. We will rerun both under our chunk budget. For LitBank and full-text JNLPBA, no reported result found on Lacuna for either dataset, so all numbers there are our own reruns.
For fairness, all systems share the same pretrained encoder, chunk budget, training data, and hyperparameter search budget. Beating Doc-ARC means exceeding our reproduced Doc-ARC on OntoNotes1000 and doing so on the cross-chunk, non-identical-string slice defined below. The reproduction should land near 84.25; if it does not, we report the gap. We will also report how far the published 84.25 is from our number.
The ablation that isolates the key ingredient keeps the two-pass memory but replaces the learned link with exact-string matching. This tests whether the learned link, rather than the second pass or the extra parameters, carries the gain.
The full-attention long-context encoder over the whole document serves as an upper reference wherever documents fit its window.
Why not just…?
Why not just use overlapping sliding windows and vote? Overlap fixes entities sliced by a boundary but brings no evidence from thousands of tokens away. Doc-ARC's +17.6 F1 on rare entities is the kind of gain that local windows cannot produce. The overlap baseline above, evaluated on the far-evidence slices, will show whether this holds in our setting.
Why not just encode the whole document with a long-context encoder? This is a real threat for documents that fit the window, and many OntoNotes1000 documents may. Two pieces of evidence argue that it does not settle the question. Flat long models still miss structure in Document Structure in Long Document Transformers, where adding structure gave +14.04 F1 on evidence selection. Full attention also degrades deep inside 30K-token contexts in Landmark Embedding. Still, if full attention matches us on long documents at comparable memory, our claim fails. The budget-sweep experiment below is designed to show exactly that if it is true.
Why not just add an off-the-shelf coreference resolver's chains as extra edges in SGT or Doc-ARC? This pipeline is the most serious alternative, because it reaches non-identical mentions without a new model. Our argument against it is the joint-labeling argument: resolver errors become permanent, the same way SGT's frozen parser errors do. That is an empirical claim, so we run this pipeline as a baseline. If it equals the joint model, the contribution reduces to "use coreference" and we will say so.
LLM prompting is a weaker alternative. GPT-4 reached only 0.28 recall on MPEA extraction in Toward Reliable Ad-hoc Scientific Information Extraction. That is a different task, so we treat it as context rather than as a baseline.
Proposed Experiments
The main experiment trains all systems on OntoNotes, evaluating on OntoNotes1000, LitBank, and full-text JNLPBA. We sweep the chunk budget $B$ over 128, 256, 512, and 1024 tokens and report entity F1 and peak memory.
The decisive analysis slices gold mentions by where their nearest same-entity mention lies, using OntoNotes coreference annotation. The three slices are: same chunk; another chunk with an identical string; and another chunk with a different string. Our claim predicts three results. Gains over Doc-ARC should be near zero on the first slice, small on the second, and largest on the third. The string-match ablation should lose most of the third-slice gain. The coreference-pipeline baseline should fall short specifically on documents where the resolver's chains are wrong.
The secondary check has two parts. On documents longer than the full-attention encoder's window, we test whether the memory model meets or beats it. On shorter documents, we measure how much of its gap to the naive chunked tagger the memory model closes per gigabyte of memory.
The claim is refuted if the third-slice gain does not exceed the second-slice gain, or if the coreference pipeline matches the joint model on the third slice.
References
- Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models
- Embedding-Based Context-Aware Reranker
- Attending to Long-Distance Document Context for Sequence Labeling
- Span Graph Transformer for Document-Level Named Entity Recognition
- A Unified Model for Opinion Target Extraction and Target Sentiment Prediction
- Knowing What, How and Why: A Near Complete Solution for Aspect-Based Sentiment Analysis
- A Joint Model for Document Segmentation and Segment Labeling
- A Truly Joint Neural Architecture for Segmentation and Parsing
- Document Structure in Long Document Transformers
- Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models
- Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Dataset