Entity Memory Instead of Full Attention: Linear-Cost Contextual Chunk Embeddings

Retrieval systems split long documents into chunks and embed each chunk separately. A chunk that says "the city's population grew" then loses the fact that "the city" is Berlin, because Berlin was named three chunks earlier. The best current fix, late chunking, runs the whole document through one long-context encoder before cutting it into chunks. It works, but its cost grows with the square of document length, and it stops working past the encoder's window. We propose a cheaper alternative. Embed chunks one at a time in a short window, and carry between them only a small memory of the entities mentioned so far: a few dozen vectors, each standing for one person, place, or company, written by a lightweight coreference model. Coreference resolution is the task of deciding which mentions ("Berlin", "the city", "it") refer to the same thing. Each new chunk reads this memory before it is embedded, so its vector knows who "it" is. The cost per chunk stays constant no matter how long the document gets.

The idea only pays off if the memory step is cheap. That step is a tiny, irregular operation: score a handful of new mentions against roughly 64 entity slots, update a few slots, and drop stale ones. Run naively, it falls below the size at which GPU parallelism helps, so launch overhead outweighs the arithmetic. Kernel fusion is the standard remedy for that. It means merging several small GPU operations into one so intermediate results never leave on-chip memory. The proposal treats coreference as the core mechanism and contextual fragmentation as the application. Parallelization overhead and fusion form the engineering constraint that decides whether the method is actually faster than the full-attention approach it replaces.

Core Claim

A coreference-supervised entity memory, passed between independently encoded chunks, can recover most of late chunking's retrieval gain on documents that fit in a long-context window. It should exceed late chunking on documents that do not fit. With cross-document batching and a single fused entity-update kernel, it should add under roughly 10% wall-clock overhead to plain independent chunking.

Memory per document is $O(Kd)$ for $K$ slots of width $d$, independent of document length $L$. Late chunking, by contrast, holds activations for all $L$ tokens and pays attention compute that grows like $L^2$. The new ingredient is entity identity as the only cross-chunk channel, which makes the method neither a generic recurrent memory nor post-hoc reranking.

Prior Work and How This Differs

Three systems motivate the gap.

Late Chunking encodes up to 8,192 tokens jointly and then mean-pools tokens into chunk vectors. It raises a context-dependent "Berlin"/"The city" similarity from 0.75 to 0.85 and gains about 3.6% relative nDCG@10 on BeIR. Its quadratic forward pass and hard window are exactly what we remove. We also reuse it as a teacher.

Embedding-Based Context-Aware Reranker (EBCAR) repairs cross-passage dependencies after retrieval, using masked attention over frozen top-20 embeddings. It cannot recover information that chunking already destroyed. We fix the embeddings at indexing time instead.

EnSI-RAG builds entity indexes with offline LLM extraction and admits this is too costly for high-churn corpora. Our entity memory is built by a small model during embedding.

Two lines of work supply the method.

Revisiting Memory-Efficient Incremental Coreference Resolution scores spans against a running set of entity vectors with gated updates. It evicts entities after fixed distances of 600 or 1,200 tokens, reaching 79.4 average F1 on OntoNotes with 2.0 GB of GPU memory. We reuse its write path. We replace fixed-distance eviction with a learned keep score, because LongtoNotes shows antecedent distances up to 11,473 tokens once documents are unsegmented, far beyond those cutoffs.

Neural Models for Reasoning over Multiple Mentions Using Coreference (Coref-GRU) shows that routing state along coreference links helps. On bAbi 1K, accuracy rises from 76.4% to 87.0%. This is evidence that explicit identity links beat soft attention alone. Our read path applies the same principle to chunk embeddings rather than recurrent states.

KVLink is the nearest architectural relative. It inserts 1 to 5 trainable link tokens between separately cached segments and reports that naive independent encoding loses up to 35% accuracy. Its link tokens carry generic, untyped information. Our memory slots carry specific, supervised entity identities. Testing whether that distinction matters is a central ablation.

Two systems results constrain the design.

SOLVING QUADRATIC PROGRAMMING PROBLEMS ON GRAPHICS PROCESSING UNIT found GPU execution slower than CPU below about 100 variables. It recovered an 11x speedup by solving 20 small problems concurrently. We adopt that "problem-parallel" pattern by batching documents at the same chunk index.

TurboTransformers shows that non-matrix-multiply operations can take up to 38% of Transformer inference time and that fusing them gives up to 2.44x over PyTorch. This is our template for fusing the entity step.

Mirage Persistent Kernel compiles whole programs into one resident GPU kernel. We treat it as a stretch option, not a dependency.

Research Program

The model starts from the same embedding encoder late chunking uses, run on chunks of about 512 tokens. Before encoding chunk $c$, the $K$ entity-slot vectors $M_{c-1}$ are projected into $K$ soft prefix tokens that the chunk tokens can attend to. This is the read path. After encoding, three things happen:

  1. A mention scorer proposes spans.
  2. Each span is pooled and scored against the slots, then either merged with a gated update or assigned a fresh slot.
  3. A learned keep score decides which slots are evicted to host memory, from which they can be recalled if a matching name reappears.

This is the write path. The chunk embedding is the mean of the chunk's token outputs, as in late chunking.

Training combines three losses:

  1. A coreference loss on the write path, using OntoNotes and LongtoNotes.
  2. A contrastive retrieval loss on FEVER and TriviaQA, matching late chunking's fine-tuning data.
  3. A distillation loss that pulls each chunk embedding toward its late-chunking counterpart, on documents short enough for the teacher.

The third loss lets the memory learn what full attention would have contributed, while the first forces that contribution through entity identity.

On the systems side, we batch $B$ documents and advance them in lockstep by chunk index. One fused kernel per step then performs span pooling, mention-to-slot scoring, gated update, and the eviction decision for all $B$ documents, with slot tensors kept contiguous per document. We profile three variants: per-document PyTorch operations, batched but unfused PyTorch, and batched plus fused. For each, we measure the entity step's share of total time as $B$, $K$, and mention density vary. This yields the granularity crossover point, which the literature characterizes for solvers and convolutional networks but not for this kind of irregular entity workload.

Baselines

Naive chunking. The do-nothing reference is the same encoder applied to independent chunks with no memory. It sets the throughput ceiling and the quality floor.

Late chunking. The strongest quality competitor is Late Chunking, run with the identical encoder at its full 8,192-token window. For documents that exceed the window, it runs over consecutive windows, which is the best available treatment without changing the method. It is the strongest baseline because it is the only method in the file that contextualizes chunk embeddings at indexing time without LLM calls. Both systems get the same fine-tuning data and the same tuning budget.

The file reports late chunking's gain on BeIR only as about 3.6% relative nDCG@10 over naive chunking. No reported result was found on Lacuna for late chunking on ConTEB, so we will measure it ourselves. Beating it means two things: within 1 nDCG@10 point on documents under 8k tokens with lower peak memory and latency, and a statistically significant gain on documents over 8k tokens.

EBCAR. The strongest post-retrieval competitor is EBCAR, which reports 64.92 average nDCG@10 on ConTEB and 29.33 queries per second on an A100. We run it on top of naive chunk embeddings and on top of ours. If our embeddings plus EBCAR beat naive embeddings plus EBCAR, the indexing-time fix adds information that reranking cannot recover.

Generic memory. The key ablation keeps the identical memory size and read path but removes coreference supervision and slot assignment, making it a generic recurrent memory in the spirit of KVLink's link tokens. It is trained with only retrieval and distillation losses. It isolates whether entity identity, rather than extra cross-chunk capacity, is what helps.

Fixed eviction. A second ablation swaps the learned keep score for the 600/1,200-token eviction rule from Revisiting Memory-Efficient Incremental Coreference Resolution. On the coreference side, that paper's 79.4 OntoNotes F1 at 2.0 GB is the reference our write path must approach.

Why not just…?

Why not just run a coreference model and textually replace pronouns with entity names before chunking? This is the most serious threat. It is cheap, needs no new architecture, and could capture much of the "the city" to "Berlin" benefit. Its weaknesses are that it carries only the name, not attributes accumulated earlier, and that it passes resolver errors straight into the text. Neural Models for Reasoning over Multiple Mentions Using Coreference notes such cascading parser errors, and A General Contextualized Rewriting Framework for Text Summarization shows rewriting without full context produces wrong anchors. We run this baseline with the same incremental coreference model. If it matches us on ConTEB, the learned read path is unnecessary and we will say so.

Why not just use late chunking with a longer-context encoder? Within the window it remains the quality target, and we expect at best parity there. Beyond the window, it must fall back to independent windows. LongtoNotes shows real antecedents span over 11,000 tokens, and its memory footprint grows with $L$. The over-8k split of the evaluation and the peak-memory curves settle where each method wins.

Why not just let a long-context LLM read the whole document, or rewrite chunks with an LLM? EnSI-RAG reports "lost in the middle" failures even for large windows. Landmark Embedding gains 8.8 LongBench points using about 2,190 tokens versus 15,500 for full context. EBCAR reports LLM rerankers at under 0.2 queries per second. These approaches answer a different cost question, and we do not compete with them on quality.

Why not just compile with an off-the-shelf fusion compiler or CUDA Graphs? Mention counts and slot assignments change every step, which Mirage Persistent Kernel identifies as the dynamic case static graphs handle poorly. Still, torch.compile with document batching is a fair contender. It is included as the batched-unfused variant. If it reaches the same overhead, the fused kernel is not a contribution.

Proposed Experiments

Retrieval quality. We evaluate on ConTEB, whose sub-datasets such as NarrativeQA, ESG Reports, and MLDR contain cross-chunk references, and on the LongEmbed tasks used by late chunking. Results are reported separately for documents under and over 8,192 tokens, with nDCG@10 as the metric. BeIR checks that general retrieval does not regress.

Coreference quality. The write path is scored on OntoNotes and LongtoNotes with average F1. Results are broken down by antecedent distance, which directly tests learned eviction against fixed eviction on reintroduced entities.

Systems costs. We report indexing throughput in documents per second, peak GPU memory, and the entity step's share of wall time across batch sizes of 1 to 64 documents, on an A100.

Decisive checks. Three outcomes decide the claim:

  1. The generic-memory ablation must trail the full model on context-dependent ConTEB queries. Otherwise coreference is decoration.
  2. The text-substitution baseline must trail the full model. Otherwise the simpler pipeline wins.
  3. The fused, batched entity step must keep total overhead over naive chunking near 10%, while late chunking's peak memory grows with length and ours stays flat.

If the first check fails but the third holds, the honest result is a fast generic chunk memory, and we would report it that way.