Attending to Memory Writes Reduces Seed Variability in Recurrent Length Generalization

Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungarXiv 2026 · arXiv

TL;DR

This paper introduces bounded delta attention, a novel mechanism that applies attention to proposed memory updates (deltas) rather than accumulated states. By constraining these updates, the authors significantly reduce performance variability across different training seeds and improve the ability of recurrent models to generalize to sequences much longer than those seen during training.

What Problem Does This Paper Address?

Recurrent Neural Networks (RNNs) are designed to track state over time, but they often struggle with length generalization—the ability to perform well on sequences longer than their training data. While adding attention mechanisms can help models learn faster, it often creates a "shortcut" problem: the model learns to rely on the attention mechanism to look back at history, neglecting to develop a robust internal recurrent update rule. Consequently, when sequences get longer than the attention window, the model fails.

Furthermore, training recurrent models is notoriously unstable. Two models trained with the same hyperparameters but different random seeds can end up with wildly different performance on long-sequence tasks. This paper seeks a way to use attention as a "scaffold" that assists the learning of recurrent updates without compromising generalization or introducing high seed-to-seed variability.

Methods

The authors propose bounded delta attention. In a standard "state attention" setup, the model attends to past hidden states. In delta attention, the model attends to the raw, uncorrected update proposals ($\delta_t$) before they are integrated into the recurrent state.

Key features include:

  • Bounded Correction: The attention mechanism produces a correction ($\epsilon_t$) that is passed through a tanh function and scaled by a small coefficient ($\alpha$). This ensures the attention mechanism can only "nudge" the recurrent update rather than completely overriding it.
  • TraceRelay Architecture: The method is implemented within TraceRelay, which uses an additive phase-based recurrence. This makes the bounded corrections interpretable as small angular shifts in a high-dimensional space.
  • Scaffold Evaluation: The authors test "OFF" versions of their models, where the attention correction is disabled at inference time to see if the recurrent part of the model learned a self-sufficient update rule.
Comparison of bounded delta attention (a) vs. standard long state-read (b).

Datasets

The study uses a controlled diagnostic task: Modulo-5 Prefix Tracking.

  • The Task: Given a sequence of integers, the model must output the cumulative sum modulo 5 at every step.
  • Training/Evaluation: Models are trained on sequences of lengths 1–40 and evaluated on lengths up to 320.
  • Significance: This task has a simple, repeatable update rule, making it an ideal "clean room" for studying how models learn to maintain and update state without the noise of natural language.

Key Insights and Ideas

  • Stability is a Choice: One of the most striking findings is that delta attention drastically reduces "seed variability." In the baseline, some seeds "get it" and others fail completely. Delta attention brings the "failing" seeds up to a much more consistent, high-performing level.
  • Placement Matters: The benefits of delta attention are most pronounced in the lower layers of the network. In these layers, standard state attention causes massive degradation in length generalization (dropping to near-chance levels), whereas delta attention remains robust.
  • Attention as a Teacher: In upper layers, the model often retains its performance even when the attention correction is turned off at test time. This suggests the attention mechanism acted as a "learning scaffold," helping the gradients flow better during training to discover a robust recurrent rule, but isn't strictly necessary for the final computation.
  • Credit Assignment vs. Forward Path: By blocking gradients through the attention inputs (the "input-SG" experiment), the authors found that the improvement in seed stability is largely driven by better credit assignment (the backward pass) rather than just the extra information provided during the forward pass.

Key Results

  • Reduced Variance: At evaluation length 80 (double the training length), the upper delta attention model achieved $70.81 \pm 8.26\%$ accuracy across seven seeds, compared to $69.21 \pm 22.42\%$ for the baseline. This represents a 63.2% reduction in standard deviation.
  • Lower Layer Superiority: In the lower layers, delta attention achieved 62.60% accuracy at length 80, while standard state attention collapsed to 19.40% (near the 20% chance level).
Performance across seven seeds at length 80 and 160. Note how delta attention (δ ON) narrows the spread compared to the Short-only baseline.

Limitations and Future Work

  • Synthetic Task: While modulo-5 is a great diagnostic, it remains to be seen how these findings translate to complex natural language tasks or next-token prediction in Large Language Models (LLMs).
  • Long-Length Decay: Even with delta attention, performance still degrades as sequences reach 320 tokens (8x training length). The "recurrent memory" problem is not fully solved for extreme extrapolations.
  • Future Directions: The authors suggest exploring this approach in language modeling objectives and investigating how multiple bounded helpers across different layers interact in more complex architectures.

Summary updated

Abstract

Recurrent memory requires learning useful updates, but attention added to assist learning can also compromise length generalization. We study bounded delta attention: causal attention over uncorrected update proposals whose bounded correction enters the recurrent update. Its implementation in TraceRelay combines additive phase recurrence with attention computed before a parallel scan. On prefix-supervised modulo-5, trained at lengths 1--40, three findings characterize its behavior at length 80. First, upper delta attention attains $70.81\pm8.26$% accuracy across seven seeds, versus $69.21\pm22.42$% for the local-attention recurrent baseline (mean $\pm$ sample SD): a 63.2% reduction in SD with a similar mean. Upper state attention also reduces variation, reaching $74.75\pm6.24$%, so this benefit is not unique to update-level attention. Disabling the upper delta correction retains $70.62\pm8.60$%. Second, blocking gradients at the delta-attention inputs changes accuracy to $65.25\pm12.40$% in the matched seven-seed comparison, increasing SD by 50.2%. It also delays a 99% validation threshold on average among the same six seeds that reach it; the 95% threshold is reached earlier on average with input-gradient blocking across all seven seeds. Third, lower delta attention exhibits substantially less extrapolation degradation than lower state attention: 62.60% versus 19.40%, against a 71.06% baseline. Removing its correction reduces accuracy to 22.56%; its advantage over lower state attention also persists under input-gradient blocking. These observations motivate bounded update-level attention as a recurrent learning scaffold, combining reduced seed variation with less severe degradation in a placement where state attention fails.
BibTeX
@misc{goo2026attending,
  author = {Sungwoo Goo and Hwi-yeol Yun and Sangkeun Jung},
  title = {{Attending to Memory Writes Reduces Seed Variability in Recurrent Length Generalization}},
  year = {2026},
  eprint = {2611.04902},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04902},
}
Figure