Write What Matters: Disentangling Token Selection from Slot Allocation in Recurrent Memory

Chen Chen, Rui ChengarXiv 2026 · arXiv

In the quest for efficient long-context AI, recurrent memory models like State Space Models (SSMs) and Linear Attention have emerged as linear-time alternatives to the quadratic-cost Transformer. However, these models face a fundamental "bottleneck" problem: they must compress an infinite stream of information into a finite set of memory slots.

A new paper, "Write What Matters: Disentangling Token Selection from Slot Allocation in Recurrent Memory," argues that we’ve been thinking about memory updates all wrong. By separating the decision of what to write from how much space to give it, the authors reveal that the "best" memory strategy depends entirely on the density of the information you're processing.

TL;DR

The authors propose Factorized Write Routing (FWR), a framework that separates the binary decision to store a token from the allocation of how many memory slots it should update. Their findings show a "density-dependent reversal": selective writing is essential for finding facts buried in noise (Passkey), but actively harmful in information-dense contexts (Feature Recall).

What Problem Does This Paper Address?

Recurrent memory models compress history into a fixed-size state. This makes every "write" operation a high-stakes gamble. If the model writes a "distractor" token (junk info), it might overwrite a crucial fact stored earlier. Conversely, if it skips a token to save space, that information is lost forever.

Existing multi-slot memories (like RAVEN) typically route every single token to a fixed number of slots ($K$). The authors argue this conflates two distinct choices:

  1. Token Selection: Is this token worth remembering at all?
  2. Slot Allocation: How much memory "budget" (how many slots) should this token get?
Figure 1: Comparison of context storage and retrieval mechanisms between existing methods and ours. Standard attention retains token-level keys and values and allows the query to attend directly to all preceding tokens, whereas Linear attention compresses the context into a shared recurrent state to alleviate the huge KV cache memory. RAVEN maintains M recurrent slots but routes every token to a fixed number K of slots. Our Factorized Write Routing (FWR) first uses a token-level gate zt to decide whether to write, and then allocates the retained token to a token-adaptive number Kt of slots. Both RAVEN and FWR read densely over all M slots, they differ in their write policies. The labels fi and di illustrate tokens that are relevant and irrelevant, respectively, to the future query q; these relevance labels are unavailable to the learned gate at inference time.

Methods

The authors introduce Factorized Write Routing (FWR). As shown in Figure 2, FWR breaks the update process into two branches:

  • The Write Gate ($z_t$): A learned binary gate that decides if a token should be written ($z_t=1$) or skipped ($z_t=0$).
  • The Slot Router ($K_t$): If a token is kept, this determines which slots to update. This can be a Fixed number of slots or an Adaptive number based on the token's importance.

Importantly, while writing is sparse (only some slots change), the Read operation remains dense—the model looks at all $M$ slots to answer a query, ensuring it doesn't need to "guess" where it stored a fact at retrieval time.

Figure 2: FWR separates token selection from slot allocation. A skipped token changes no slot; a kept token updates either a fixed or token-adaptive number of slots. Reading remains dense, so a query need not predict the write-time slot identity.

Datasets

The study uses two synthetic tasks designed to test different information densities:

  1. Feature Recall (High Density): Every token in the context is a unique key-value pair that could be queried later. There are no distractors.
  2. Passkey (Low Density): A few important facts are hidden among a sea of "filler" tokens (75% of the context is noise).
Summary of the two synthetic tasks.

Key Insights and Ideas

  • The Density-Dependent Reversal: There is no "universal" best routing strategy. In the dense Feature Recall task, the best strategy was to always write to an intermediate number of slots. In the sparse Passkey task, selective writing (learned gating) was the clear winner.
  • Sparse Writing is a Double-Edged Sword: While sparse writing reduces memory interference, it introduces the risk of "false negatives"—accidentally skipping a token that turns out to be important later.
  • Adaptive $K$ isn't enough: Simply letting the model choose how many slots to update (adaptive $K$) performed worse than using a fixed $K$ combined with a learned "skip/keep" gate.
  • The Oracle Gap: By testing an "Oracle" gate (which knows exactly which tokens are facts), the authors showed there is significant "headroom" for improvement. Current learned gates are good, but they still struggle to perfectly distinguish facts from distractors.

Key Results

The quantitative results highlight the stark difference between the two regimes:

  • On Passkey (Sparse): Models using the learned FWR gate achieved 0.2216 accuracy, dwarfing the "always-write" RAVEN baseline, which struggled at 0.0475.
  • On Feature Recall (Dense): The learned gate actually hurt performance. The best deployable model was a fixed-allocation RAVEN ($K=4$) at 0.4716, while the FWR variants lagged behind.
  • Fixed vs. Dense: In high-density tasks, writing to all 16 slots (Dense) significantly outperformed writing to only 2 slots (Fixed $K=2$), proving that small fixed $K$ values can under-allocate memory when every token matters.
Table 3: Ten-seed mechanism comparison on LONGMEAN. Non-deterministic entries report mean ± standard deviation across matched seeds 1–10. Hyperparameters are fixed before test evaluation. Bold marks the highest deployable mean per task. Oracle methods (†) use ground-truth token-type labels and are diagnostic. Paired inference for five pre-specified contrasts appears in Appendix Table 7.

Limitations and Future Work

The authors identify several areas for further exploration:

  • Logical vs. Physical Sparsity: Currently, the "sparse" routing is implemented using dense tensors, meaning it doesn't yet provide a wall-clock speedup. Realizing these gains requires specialized sparse kernels.
  • Synthetic to Natural: The study relies on synthetic tasks. Future work needs to validate these findings on natural language, where "relevance" is often ambiguous rather than binary.
  • Query-Conditioned Gating: The current gate only looks at the token itself. A more powerful gate might look at the current memory state or the likely future queries to decide what to store.

By disentangling selection from allocation, this paper provides a roadmap for designing more robust recurrent memories that can adapt to the "signal-to-noise" ratio of the data they process.

Summary updated

Abstract

Long-sequence models must retain earlier information. Full attention keeps all past tokens but grows expensive, and recurrent-memory models compress history into fixed slots for linear-time processing. Limited capacity makes writes consequential: distractors can interfere with useful content, while skipped evidence cannot be recalled. Existing multi-slot memories write every token to the same number of slots, merging whether to store it with how many slots to update. We introduce Factorized Write Routing (FWR), which first keeps or skips each token and then chooses a fixed or token-adaptive number of slots. Queries still read all slots. Across ten seeds on two synthetic tasks that query facts after long contexts, we observe a density-dependent reversal. With four slots per write, long-context accuracy favors always writing on dense Feature Recall (0.472 vs 0.337), but learned selective writing on sparse-fact Passkey (0.222 vs 0.048). The tested adaptive rule underperforms fixed allocation under the learned gate on both tasks. A diagnostic oracle given fact labels unavailable at deployment reaches 0.372 on Passkey, showing headroom in filtering distractors. The results suggest avoiding missed writes in information-dense contexts and filtering distractors when they dominate.
BibTeX
@misc{chen2026write,
  author = {Chen Chen and Rui Cheng},
  title = {{Write What Matters: Disentangling Token Selection from Slot Allocation in Recurrent Memory}},
  year = {2026},
  eprint = {2610.34180},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34180},
}
Figure