Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

Ali Hatamizadeh, Yejin Choi, Jan KautzarXiv 2026 · arXiv

TL;DR

Gated DeltaNet-2 advances linear recurrent attention by breaking the traditional scalar tie between erasing old memories and writing new ones. By introducing decoupled, channel-wise erase and write gates, it significantly reduces memory interference in fixed-size recurrent states. This architecture achieves state-of-the-art performance on long-context retrieval and language modeling benchmarks while maintaining highly efficient parallel training throughput.

What Problem Does This Paper Address?

While Transformers dominate modern AI, their quadratic computational complexity ($O(L^2)$ with respect to sequence length $L$) makes long-context processing and high-throughput inference incredibly expensive. Linear recurrent attention models (like Mamba or DeltaNet) offer a solution by compressing history into a fixed-size recurrent state, achieving linear scaling ($O(L)$).

However, compressing a massive context into a finite memory space introduces a major challenge: interference. How do you update this compressed memory with new information without scrambling existing, valid associations?

"Delta-rule" models attempt to solve this by reading the memory at the current key location and subtracting it before writing a new value. Yet, prior state-of-the-art models like Gated DeltaNet and Kimi Delta Attention (KDA) suffered from a shared limitation: they used a single scalar gate to control both how much old content to erase (on the key side) and how much new content to commit (on the value side). Because erasing and writing act on completely different axes of the memory state, tying them to a single scalar acts as a severe modeling bottleneck.

Methods

The authors introduce Gated DeltaNet-2, which decouples the erase and write operations using two separate channel-wise gates:

  1. Channel-wise Erase Gate ($b_t$): Operates on the key axis, determining exactly which coordinates of the old memory should be cleared.
  2. Channel-wise Write Gate ($w_t$): Operates on the value axis, deciding which coordinates of the incoming value should be committed to the state.

Mathematically, the Gated Delta Rule-2 update is formulated as: $$S_t = (I - k_t(b_t \odot k_t)^\top) D_t S_{t-1} + k_t(w_t \odot v_t)^\top$$ where $D_t$ represents the channel-wise decay matrix inherited from KDA.

Visualization of the hybrid architecture and block design of Gated DeltaNet-2.

To ensure this model can be trained efficiently on modern hardware, the authors derived a chunkwise WY algorithm. By absorbing the cumulative channel-wise decay into asymmetric rank-one erase factors, they formulated a gate-aware parallel backward pass. This allows the model to utilize dense matrix multiplications inside fixed-size chunks, mapping perfectly to GPU tensor cores via custom Triton kernels.

The authors evaluate Gated DeltaNet-2 in two structural flavors: a recurrent-only model and a hybrid model that pairs the recurrent layer with Sliding-Window Attention (SWA) to handle exact local interactions.

Datasets

The models were pretrained from scratch on 100B tokens from the FineWeb-Edu dataset.

Evaluation was performed across a diverse suite of standard benchmarks:

  • Language Modeling: WikiText and LAMBADA (perplexity and zero-shot accuracy).
  • Commonsense Reasoning: A 9-task suite including PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, OpenBookQA, Social IQa, and BoolQ.
  • Synthetic Long-Context Retrieval: The RULER benchmark, specifically testing Single Needle-In-A-Haystack (S-NIAH) and Multi-Key Needle-In-A-Haystack (MK-NIAH) tasks up to 8K context lengths.
  • Real-World Retrieval: A recall-heavy suite spanning SWDE (HTML extraction), FDA (PDF key-value retrieval), SQuAD, TriviaQA, Natural Questions (NQ), and DROP.

Key Insights and Ideas

  • The Asymmetry of Memory Editing: Erasing is fundamentally a key-side operation (identifying where to clear), while writing is a value-side operation (identifying what features to store). Decoupling them allows the model to clear broad contexts via decay, wipe out specific stale associations via erase, and isolate specific value channels to persist via write.
  • The Erase Gate is the MVP: Through gate structure ablations, the authors discovered that the channel-wise erase gate ($b_t$) contributes the majority of the performance gains compared to the write gate ($w_t$). Selectively protecting or revising key channels is more critical for preventing interference than reweighting written values.
  • Hardware Efficiency Preserved: Introducing complex, channel-wise gating mechanisms often breaks the parallelizability of RNNs. Gated DeltaNet-2 bypasses this by sharing the same lower-triangular WY inverse matrix across both the erase-side and write-side auxiliary computations, keeping training fast.

Key Results

At a matched 1.3B parameter scale, Gated DeltaNet-2 achieved the strongest overall results against Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants.

  • Long-Context Mastery: On the interference-heavy Multi-Key Needle-In-A-Haystack task (MK-NIAH-1) at a 4K context length, the recurrent Gated DeltaNet-2 scored 37.8%, vastly outperforming KDA (28.0%) and Mamba-2 (21.4%).
  • Real-World Retrieval: In hybrid configurations, Gated DeltaNet-2 achieved the highest average accuracy (42.28%) across real-world tasks like SQuAD and TriviaQA.
  • Flat Throughput Scaling: As shown in Figure 2, Gated DeltaNet-2 maintains a highly efficient, near-flat training throughput profile on an H100 GPU as sequence length scales from 2K to 16K, whereas standard Transformers suffer a sharp performance degradation.
Training throughput on a H100 GPU.

Limitations and Future Work

  • Erase Range Constraints: The authors experimented with expanding the erase gate range from $[0, 1]^{d_k}$ to $[0, 2]^{d_k}$ to allow for over-erasing (negative eigenvalues), but found it provided no consistent gains at the 1.3B parameter scale.
  • Scale Limitations: The evaluation is bounded to a 1.3B parameter model trained on 100B tokens. Testing how well these decoupled gates scale to larger parameter sizes (e.g., 7B+) and trillion-token regimes remains an area for future exploration.

Summary updated

Abstract

Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.
BibTeX
@misc{hatamizadeh2026gated,
  author = {Ali Hatamizadeh and Yejin Choi and Jan Kautz},
  title = {{Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention}},
  year = {2026},
  eprint = {2605.22791},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.22791},
}

Loading related content...

Figure