Admitting Entities to Memory at First Mention: Grammatical Prominence as a Cache Policy for Book-Length Coreference

Coreference resolution means deciding which mentions in a text ("Napoleon", "the pig", "he") refer to the same entity. On book-length text, modern systems cannot compare every mention with every earlier one, so they keep a small memory, or cache, of candidate entities. The cache has to decide what to forget. The strongest current design counts how often each entity has appeared and keeps the frequent ones. That design has a blind spot. An entity that will become a central character looks exactly like a throwaway mention the first time it appears, and it only earns a protected place after it has been mentioned several times. If it then vanishes for a few chapters, it may already have been evicted.

Linguistics offers a cheap signal that is available immediately. Entities introduced as grammatical subjects or agents tend to be mentioned again. Entities introduced in oblique positions, or under negation ("no car was found") or modality ("a car might be there"), usually die after one mention. We propose a cache that scores each entity at the moment it is introduced, using this prominence information. The cache then combines that score with frequency evidence as the document unfolds. We test whether this beats frequency-only caching on literary and long-document coreference. We also test whether the gain comes from explicit grammatical structure or from what a learned encoder already knows about its input.

Core Claim

Frequency-based eviction is reactive: it learns that an entity matters only after the entity has proved it. A cache that predicts how long an entity will live, scored from the clause that first introduces it, should keep future main characters through topic switches before frequency evidence has built up. It should also throw out the many one-off mentions that currently compete for cache slots. The first half of the claim is the cache policy. The second half is an empirical question this note treats as open: whether grammatical role and clausal environment at introduction still predict re-mention at distances of thousands of tokens, a regime the classic findings never measured.

Grounding in Prior Work

Two papers motivate the problem. Dual Cache for Long Document Neural Coreference Resolution splits memory into a recency-based local cache and a frequency-based global cache, promoting entities once they cross a frequency threshold. It finds that 6% of entities cause 97% of cache misses under recency-only caching. The authors note that their design is not grounded in theories of discourse salience. Our proposal keeps their two-tier structure and changes only the rule for promotion and eviction. The Life and Death of Discourse Entities shows that about 56% of OntoNotes noun phrases are singletons, meaning entities mentioned only once. It also shows that grammatical role plus negation and modality scope separate singletons from coreferent mentions with 78% accuracy. That result was measured on documents averaging 466 tokens, not on books. We use it as the source of the predictive features.

Three papers supply method or precedent. Capturing Salience with a Trainable Cache Model for Zero-anaphora Resolution already trained a ranker over Japanese case markers to decide which entities stay in a 15-slot cache, reaching 85.8% antecedent coverage. That ranker was a support-vector model on short news text. We revive the idea of a learned, salience-driven cache inside a neural long-document model and evaluate it at book scale. Entity-Centric Joint Modeling of Japanese Coreference Resolution and Predicate Argument Structure Analysis shows that tracking who-did-what-to-whom (predicate-argument structure) sharply improves resolution of omitted pronouns across sentences. This supports using agent-versus-patient roles alongside surface subjecthood. Focusing and Reference Resolution in PUNDIT argues that surface subject and object are steadier focus cues than deep thematic roles, for example across passive voice. We therefore include both feature families and let the ablations decide between them.

Two papers constrain the design. Structural Guidance for Transformer Language Models found that hard attention masks built from parse trees gave no consistent benefit over unmasked training. For that reason we use prominence only to rank cache entries, never to block attention. Bridging Context Gaps runs coreference offline and rewrites text before a large model reads it. It shows the value of entity tracking for long-context question answering, and it also shows how upstream errors propagate through the rewritten text. Our approach tolerates such errors more gracefully: a wrong prominence score only changes what is retained, and later frequency evidence can override it. G-KV scores tokens for eviction by attention accumulated over time. It is the closest "learned importance" analogue, but it is reactive in the same way frequency counting is, and it has no notion of entities.

Research Program

The system is a cache-based incremental coreference model in the style of Dual Cache, built on a SpanBERT-type encoder (SpanBERT). The change is a retention score for each cached entity $e$ at document position $t$:

$$ s(e,t) = \sigma\big(w^\top \phi(m_e^{1}) + \beta \log(1+c_e(t)) - \lambda\, d_e(t)\big) $$

Here $\phi(m_e^{1})$ is a feature vector for the entity's first mention. It contains surface grammatical role, predicate-argument role, sentence-initial position, negation/modality scope, and whether the mention is a proper name or a definite or indefinite noun phrase. $c_e(t)$ is the number of mentions so far, $d_e(t)$ is the token distance since the last mention, and $w, \beta, \lambda$ are learned. When a slot is needed, the cache evicts the lowest-scoring entity. The score is trained to predict whether $e$ will be mentioned again within a horizon $H$ tokens, with $H$ varied from 500 to 10,000. Its labels come for free from gold coreference chains. The frequency and distance terms let the model override a wrong prior once an entity's actual behavior is observed. This addresses the error cascade that sank the old greedy salience caches.

We build three variants of $\phi$. The syntactic variant uses features from an off-the-shelf dependency parser and semantic role labeler. The learned variant uses only the encoder's span embedding of the first mention, passed through a small network. The combined variant uses both. The analysis half of the program asks how the predictive power of each feature decays as the horizon $H$ grows. This tests on LongtoNotes and LitBank whether the hierarchy that held at 466 tokens survives at the 11,473-token antecedent distances that Longtonotes restored by merging split documents.

Baselines

The naive reference is a recency-only (least-recently-used) cache with the same encoder and slot budget. This is the failure mode documented in Dual Cache, where performance collapses once mention spread exceeds about 1,000 words. A second reference is a non-cached model that scores mentions against all earlier ones, as an upper bound on accuracy that ignores cost.

The strongest existing method is Dual Cache itself. It reports 79.5 F1 on LitBank, and on the 30,000-word Animal Farm test it beats its baseline by 10.5 F1 while using 70% less inference time (Dual Cache). It is the right competitor because it already solves topic switching with the same two-tier memory we use, so any gain we show isolates the promotion rule. Since no official code is indexed, we will reimplement it in our codebase with the same encoder, training data, cache sizes, and hyperparameter search budget. We report both the published 79.5 and our reproduced number. Beating it means a CoNLL-average F1 gain on LitBank and LongtoNotes at equal cache size, or equal F1 at a strictly smaller cache. Each claim requires confidence intervals across at least three seeds. On LongtoNotes we found no reported cache-model result on Lacuna; the only figure is a +1% gain for SpanBERT on documents over 2,000 tokens when trained on LongtoNotes (Longtonotes).

The key ablations isolate the two ingredients. The first sets $w=0$, which yields a learned frequency-plus-recency cache, testing whether the first-mention prior matters at all. The second compares the syntactic, learned, and combined versions of $\phi$, testing whether explicit grammar matters beyond the encoder's own knowledge. We also include an attention-accumulation entity score in the spirit of G-KV, summed over each entity's mention spans.

Why not just…?

Why not just make the cache bigger? A large enough cache makes eviction policy irrelevant. The question is what accuracy is achievable at a fixed memory and time budget, and Dual Cache's speed gains come precisely from small caches. We will plot F1 against cache size for every policy. If our curve does not dominate frequency-only caching at small and medium sizes and merely converge at large ones, the proposal fails on its own terms.

Why not just lower the frequency threshold in Dual Cache? Promoting after one or two mentions instead of more is the obvious tweak. Because roughly 56% of noun phrases are singletons, a low threshold should fill the global cache with entities that never return (Life and Death of Discourse Entities). We tune the threshold over its full range within the same budget as our model, so this baseline gets its best setting.

Why not just learn the lifespan predictor from span embeddings and skip the parser? This is the real threat. A pretrained encoder may already represent subjecthood and negation scope, in which case the parser adds latency and nothing else. The learned-$\phi$ variant is therefore not an ablation of convenience but a co-equal hypothesis. If it matches the combined variant, we report that the predictive admission rule is the contribution and grammatical prominence is something the encoder has internalized. We would then confirm this with linear probes of the span embeddings for role and scope. Explicit syntax would earn its place only if it helps on out-of-domain or noisy text, or at long horizons where encoder features may not reach.

Why not just rewrite the text with an offline resolver, as LQCA does? That approach targets a different consumer, a frozen large model, and still needs an underlying coreference system that handles book-length input. Our cache is exactly such a component, so the two are complementary rather than competing.

Proposed Experiments

The first check is cheap and decisive, and runs before any model is built. On LitBank, we take the small set of entities that cause most cache misses under recency caching and ask whether their first mentions are disproportionately subjects or agents, outside negation and modality scope. We compare against the base rate for all entities. If prominence at introduction does not separate these entities from the rest, the core claim is dead and we stop.

The second experiment is the horizon analysis on LongtoNotes and LitBank. It measures the area under the ROC curve for predicting re-mention within $H$ tokens, for each feature group, as $H$ grows to 10,000. This shows whether the hierarchy decays, persists, or gives way to thematic roles at long range, a question the Longtonotes data makes answerable for the first time.

The third is the end-to-end comparison against all baselines above. It reports CoNLL-average F1 on LitBank, LongtoNotes (with the broadcast-conversation genre, averaging 4,071 tokens, broken out), and Animal Farm, along with cache-miss rates and wall-clock time. We expect the gain to concentrate on entities that reappear after long gaps early in their lifetimes. We will report F1 stratified by gap length and by mention count at the time of the gap, which is where a reactive frequency policy should lose and a predictive one should win.