Decoupling Activation Memory from Sequence Length

44 concepts33 papers1 proposal

When training deep learning models on long sequences, researchers inevitably hit a "memory wall." This bottleneck is rarely caused by the model's parameters; instead, it is driven by activation memory—the intermediate layer outputs that must be cached during the forward pass so that gradients can be computed during the backward pass.

For a sequence of length $L$, storing these activations scales at least linearly ($O(L)$), and quadratically ($O(L^2)$) for standard self-attention. As context lengths push into millions of tokens, storing the full activation history quickly exceeds hardware capacity.

To break this dependency, a powerful paradigm has emerged: decoupling activation memory from sequence length. By combining chunk-wise sequential processing with local activation recomputation, models can process arbitrarily long sequences while maintaining a flat, constant memory footprint ($O(1)$) relative to the sequence length.


The Core Mechanism: Chunk-Wise Processing

The foundation of constant-memory sequence processing is avoiding the monolithic execution of long sequences. Instead, an extremely long sequence is partitioned into smaller, fixed-size segments or chunks.

Monolithic Processing (High Memory):
[======================= Full Sequence L =======================] -> Cache all activations

Chunk-Wise Processing (Constant Memory):
[ Chunk 1 ] ---- State ----> [ Chunk 2 ] ---- State ----> [ Chunk 3 ]
(Cache C)                    (Cache C)                    (Cache C)

To preserve global context across these boundaries, models pass a compact, fixed-size hidden state or memory token from one chunk to the next.

  • Xiulong Yuan et al. (2025) demonstrate this in ChunkFlow, where managing state transitions between fixed-size segments allows models to be fine-tuned on massive contexts using the exact same memory footprint required for short sequences.
  • Aydar Bulatov et al. (2024) implement this via the Recurrent Memory Transformer (RMT), which passes a small set of dedicated memory tokens recurrently between segments, enabling standard Transformers to process millions of tokens with linear computational scaling.
  • This chunking behavior extends naturally to long-form reasoning. Milad Aghajohari et al. (2026) introduce a "Markovian Thinking Paradigm" that partitions extended reasoning chains into discrete segments, preserving only a fixed-size suffix of the previous segment to prevent quadratic memory growth during generation.

The Backward Pass: Recomputation and Staged Backpropagation

While chunking keeps the forward pass memory-efficient, standard Backpropagation Through Time (BPTT) would still require caching the activations of every chunk to compute gradients at the very end. To achieve true memory-length decoupling, the system must alter how gradients are calculated.

Instead of keeping the entire activation history in GPU memory, the system stores only the compressed hidden states or boundary tokens at the edges of each chunk. During the backward pass, the system processes the chunks in reverse order. For each chunk, it uses the saved boundary state to re-run the forward pass locally for that chunk alone, reconstructing the intermediate activations on the fly, computing the gradients, and then immediately discarding those activations.

Forward Pass:  [Chunk 1] --save state--> [Chunk 2] --save state--> [Chunk 3]
                                                                        |
Backward Pass: [Chunk 1] <--recompute-- [Chunk 2] <--recompute-- [Chunk 3]
               (Gradients)               (Gradients)               (Gradients)

This classic time-memory tradeoff—performing a local forward pass twice to avoid storing activations globally—flattens the training memory curve.

  • Staged Backpropagation: Early foundations were laid by Wang Ling et al. (2017), who utilized staged back-propagation to break long program-generation sequences into discrete chunks while maintaining global hidden states across segments.
  • Memory Replay: The authors of Memformer (2022) formalize this as Memory Replay Back-Propagation (MRBP), demonstrating that reconstructing local computational graphs during the backward pass dramatically reduces training memory with minimal impact on processing speed.
  • Modern LLM Scaling: Weigao Sun et al. (2025) leverage this in Sequence Accumulation (SA), carrying over a fixed-size recurrent state between sub-sequences and recomputing activations locally to train linear sequence models on million-token contexts on a single GPU. Similarly, Wenhao Li et al. (2026) show that chunk-wise training with on-the-fly recomputation reduces activation memory to a constant footprint, decoupling training costs from total context length.

Generalization Across Architectures and Domains

The strategy of chunking and selective recomputation is architecture-agnostic and has been adapted across various models to bypass sequence-length limitations.

Linear Attention and Transformers

Standard attention blocks can be refactored to avoid materializing large matrices. Leo Feng et al. (2023) introduce the Constant Memory Attention Block (CMAB), which updates its state in constant time and memory per new data point. For Linear Transformers, Krzysztof Marcin Choromanski et al. (2021) utilize the mathematical structure of kernel-based attention to store only cumulative boundary states, reducing Performer training memory to a sub-linear or constant footprint. Even within standard feed-forward layers, Nikita Kitaev et al. (2020) show in the Reformer that partitioning large intermediate computations into independent blocks along the sequence dimension significantly lowers peak memory.

Computer Vision and Diffusion Models

The memory penalty of sequential processing also plagues non-text domains. In video processing, Bo Pang et al. (2021) decompose the forward and backward propagation of 3D convolutional networks into serial steps with truncated gradients, allowing models to maintain semantic continuity over long videos without exceeding GPU limits. In generative modeling, Mintong Kang et al. (2023) apply segment-wise backpropagation to the long sequential reverse-steps of diffusion models, caching intermediate samples to achieve constant memory complexity during adversarial gradient calculations.

Stochastic Truncation

When exact gradients across the entire sequence are not strictly necessary, chunking can be paired with truncation to save additional overhead. Jiaqing Zhang et al. (2025) use randomly sampled windows of training history to calculate gradients for long-sequence dataset distillation, capturing long-term dependencies while maintaining a manageable footprint. Similarly, Cees Snoek (2022) demonstrates that restricting gradient calculations to specific windows of iterations allows for deep recurrent reasoning chains while maintaining a constant memory profile during backpropagation.

Key Foundational Work

Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems (2017) — contributed 2 concept elements to this direction.
Iterative Joint Image Demosaicking and Denoising Using a Residual Denoising Network (2019) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_memory_length_decoupling_76eb3ffe,
  title = {memory-length decoupling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/memory-length-decoupling/rdr_76eb3ffee47b4c7a8073198b3f8f05bc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_memory_length_decoupling_76eb3ffe_v1,
  title = {memory-length decoupling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/memory-length-decoupling/rdr_76eb3ffee47b4c7a8073198b3f8f05bc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake