Silent Memory for Sparse Long-Story Decoding

When a language model writes a 10,000-word story, every new word looks back at every earlier word. The model stores a running record of those earlier words, called the key-value cache, and it grows with the text. That makes long generation slow and memory-hungry. The usual fix is to throw part of the record away, keeping only recent text or text the model currently seems to need. Stories break this assumption. A character introduced in chapter one and absent for eight thousand words must come back with the same name, age, and grudges. If her part of the record was deleted while she was offstage, the model cannot recover her. We propose a different policy, borrowed from how the brain handles temporarily irrelevant information. Past passages that are not currently needed stop taking part in attention and leave fast memory, but they are never deleted. Each passage keeps one small summary vector on the GPU. When that vector shows the passage has become relevant again, its full record is fetched back.

Neuroscience supplies the motivation. Monkey prefrontal cortex and task-trained recurrent networks keep temporarily irrelevant information in "activity-silent" states, meaning it is stored without active firing and can be reactivated later (Effects of noise and metabolic cost on cortical task representations). Human frontoparietal cortex also turns down coding for rules once they become routine (Adaptive Coding of Task-Relevant Information in Human Frontoparietal Cortex). The engineering question is whether the same pattern, sparse active computation with a silent but recoverable store, lets long-form generation run with a small fraction of attention and GPU memory while avoiding the character and scene errors that ordinary eviction causes. A secondary question is whether moderate sparsity actually reduces narrative drift by cutting attention to stale text.

Core Claim

The closest existing system is Dynamic Hierarchical Sparse Attention (DHSA) (Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference). DHSA uses a small trained network to split the input at natural sentence and paragraph boundaries and pools each chunk into one representative vector. It first scores chunks against the current query, then attends token by token only inside the top-scoring chunks. It recovers over 95% of dense attention mass using about 12% of tokens, and it beats fixed block-sparse attention by 12–20% relative accuracy on LongBench. However, DHSA is tested only on reading a long input (prefill). Neither it nor anything else in our file examines generation, where the chunks are being written as the model goes, or checks whether sparse routing damages or protects narrative coherence.

Our claim has two parts. First, when chunk-level sparse attention is carried into generation, the coherence errors it causes are not spread evenly. They should cluster at re-entries, that is, when a character, object, or location returns after a long absence. Second, a policy that keeps a permanent chunk-level trace of every past chunk and reactivates the full token record on demand will remove most of those errors at the same GPU memory as eviction. Existing eviction methods, such as the anchor-projection method in LLM-AnDPro, discard token records permanently, so they cannot reactivate anything. That difference, reactivation versus deletion, is the proposal's key ingredient, and the experiments are built to isolate it.

Research Program

We will build the decoding extension on the official DHSA codebase and apply it to LongWriter-9B, the open model from LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs.

During generation, the boundary predictor runs on the growing output and closes a chunk when it detects a sentence or paragraph end. Each closed chunk gets its length-normalized representative vector, as in DHSA. These representatives always stay on the GPU. They are small, a single vector per chunk per layer, so their cost grows with the number of chunks rather than the number of tokens.

The records for individual tokens follow a three-tier policy. Text within a fixed recent window is always kept on the GPU, fully active. Older chunks are "silent": their token records move to CPU memory and are left out of attention. At each step, or every few steps to spread out transfer cost, the current query scores all chunk representatives. Chunks that rise into the top-k are prefetched back to the GPU, and token-level attention runs inside them under a fixed token budget.

The cost of reactivation is transfer latency, not lost information. We will measure that latency directly rather than assume it away. The file raises a hardware-algorithm gap: brains maintain silent states at almost no cost, while ordinary matrix hardware gains nothing from inactive units. Here the saving comes from shrinking the working set held on the GPU, not from skipped multiplications.

A second arm tests adaptive budgets. The analogy is the neural efficiency finding that connectivity between regions drops as a skill becomes automatic (Parallel Alterations of Functional Connectivity during Execution and Imagination after Motor Imagery Learning). We will let the token budget shrink once a scene is established, detected by stable chunk-routing entropy, and expand it at detected scene boundaries.

Two cautions from the same literature shape this arm. Zhang et al. found no linear relation between how much connectivity dropped and how much performance improved, so we do not assume more sparsity means better stories. Instead we sweep density and report the full curve. Interpretable multimodal fusion networks reveal mechanisms of brain cognition links dense, noisy connectivity to lower cognitive performance. That motivates the falsifiable secondary hypothesis that moderate sparsity lowers repetition and drift compared with dense attention. We treat it as an open empirical question, not a premise.

Coherence measurement needs care. Holistic human ratings of coherence agree poorly: Krippendorff's α was 0.19 in SNaC: Coherence Error Detection for Narrative Summarization. SNaC's span-level categories reached 0.69 on character errors. We will therefore annotate generated stories with the SNaC categories, focusing on character errors, scene-transition errors, inconsistencies, and repetitions.

We will also train a detector on SNaC. The reported T5 detector recall is 0.90 on character errors and 0.82 on scene transitions. SNaC covers summaries of narratives, not stories, so the detector serves only as a scalable screen, calibrated against a human-annotated subset. It is not the final judge.

We will add a controlled probe we call re-entry tests. Prompts from LongBench-Write and WritingPrompts are extended with outline constraints: a character with stated attributes is introduced early and must reappear after a specified gap of 2,000, 5,000, or 10,000 words. We then check attribute consistency at the return, both by string-matched facts and by annotator verification. The probe directly measures what eviction should break and reactivation should fix.

Baselines

The naive reference is dense attention on the same LongWriter-9B checkpoint with the same prompts and the same sampling settings. It sets the coherence ceiling that sparse methods should approach. It also sets the memory and latency cost we aim to cut. On quality, the anchor is LongWriter-9B-DPO's reported overall score of 84.0 on LongBench-Write (LongWriter). We will re-run it in our own harness, because judge-based scores shift with setup, and report both our reproduction and the paper's number. A sparse method "matches dense" if its LongBench-Write score falls within judge noise of the reproduced dense score, estimated by rerunning the judge, at 12.5% or lower attention density.

Fixed sparsity is the cheapest competitor. We will run two versions: a sliding window with a few permanently kept initial tokens, and fixed-stride block-sparse attention at the same density. Both are fast and simple. DHSA's reported 12–20% relative gain over block-sparse attention was measured in prefill, and whether it carries over to generation is part of what we test. To beat these baselines, our method must show fewer character and scene-transition errors per thousand words and higher re-entry accuracy at equal GPU memory.

The strongest existing efficiency competitor is permanent KV eviction. We will use LLM-AnDPro from its released code, given the same GPU-resident token budget our method keeps active. For this baseline, no reported result on long-form narrative generation was found on Lacuna. It is the right competitor because it is content-aware and already built for decoding.

The ablation that isolates our key ingredient is our own system with reactivation switched off. It uses the same boundaries, the same chunk routing, and the same budget, but silent chunks are deleted instead of offloaded. Any gap between this ablation and the full method is attributable to recoverable storage alone. We will also run the ablation with DHSA's original fixed-length chunks, which isolates the contribution of semantic boundaries.

The strongest coherence competitor works outside the model. It is a Re3-style pipeline (Re3: Generating Longer Stories With Recursive Reprompting and Revision) in which a rolling window is supplemented by an outline, a character attribute dictionary, and running summaries. Re3 reported a 14% absolute gain in overall plot coherence and a 20% absolute gain in premise relevance over rolling-window GPT-3. We will implement it on LongWriter-9B with its context capped at the same number of active tokens as our method. We will count its extra summarization and planning calls toward total compute.

Why not just…?

Why not just keep dense attention and buy memory, or quantize? Dense attention is the quality ceiling, and if memory were free there would be nothing to fix. But record size grows linearly with length and attention cost grows quadratically. At 10,000+ words and beyond, which LongWriter targets, that cost sets the limit on batch size and length. We report quality per GPU-gigabyte and per second, so the reader can see exactly what sparsity costs in quality. The secondary hypothesis that sparsity improves coherence is included because, if true, it would change this trade-off. We expect it may well fail and will report it either way.

Why not just use an existing KV eviction method? This is the most direct competitor, and it may be good enough for most text. Eviction assumes that low attention now predicts low need later. In a story, a character offstage for chapters is exactly the kind of text with low current attention and high future need. The re-entry tests at increasing gaps settle the question. If eviction's re-entry accuracy stays level with ours as the gap grows, the reactivation idea is unnecessary and we will say so.

Why not just offload memory into prompts, as Re3 and AgentWrite do? This is a real threat. External summaries are cheap, need no changes to the model, and have documented coherence gains. Their weaknesses are also documented. Re3's editor mostly handles surface character attributes and struggles with spatial and temporal logic. LongWriter reports that section-by-section generation reads mechanically at section breaks, and that generating sections in parallel costs 6% in coherence. A summary is lossy at the moment it is written. A silent token record is not lossy, only inactive. We run the Re3-style pipeline under matched active context and total compute, and we also test the combination of both approaches. If the prompt pipeline alone matches our re-entry accuracy and error rates, the internal mechanism does not earn its complexity.

Why not just let RL-trained planning handle it? LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning shows that intermediate thinking tokens are critical. Removing them dropped Arena-Write Elo from 1221 to 668. It also reports a saturation point beyond which extra thinking uses up context without improving writing. Our method is orthogonal to planning, not a replacement: it reduces what the context has to hold. If time permits, we will apply it to generation from a thinking-enabled model to see whether offloading pushes the saturation point further out.

Proposed Experiments and Decisive Checks

The main experiment uses LongBench-Write (120 prompts, English and Chinese) and WritingBench, plus about 200 WritingPrompts premises with re-entry constraints, generating stories of 2,000 to 16,000 words. Every method from the Baselines section is swept across attention densities of 6.25%, 12.5%, and 25%.

We report four groups of measurements. The first is LongBench-Write and WritingBench scores. The second is repetition rates (rep-8 and rep-16, as in DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer). The third is SNaC-category error rates per thousand words, from the detector and from a human-annotated subset of about 60 stories per condition. The fourth is re-entry attribute accuracy as a function of gap length. Alongside these we report peak GPU memory, tokens per second, and CPU-to-GPU transfer volume.

Three outcomes are decisive. First, the error-location analysis: under the no-reactivation ablation, a disproportionate share of character and scene errors should occur at re-entry points. If errors are instead spread evenly, the core diagnosis is wrong. Second, the gap-length curve: eviction's re-entry accuracy should decay with gap length while the full method's stays near dense. If both decay alike, the chunk representatives are failing to signal when a chunk is needed again. Chunk-level routing recall against dense attention mass during generation, which we log directly, will distinguish that failure from others. Third, the matched-compute comparison against the Re3-style pipeline decides whether internal silent memory adds anything over external summaries.

The adaptive-budget arm and the sparsity-improves-coherence hypothesis are secondary. They are reported as density-versus-quality curves, with no claim made unless the curve dips below the dense baseline's error rate by more than annotation noise.