TL;DR
Gated DeltaNet-2 advances linear recurrent attention by breaking the traditional scalar tie between erasing old memories and writing new ones. By introducing decoupled, channel-wise erase and write gates, it significantly reduces memory interference in fixed-size recurrent states. This architecture achieves state-of-the-art performance on long-context retrieval and language modeling benchmarks while maintaining highly efficient parallel training throughput.
What Problem Does This Paper Address?
While Transformers dominate modern AI, their quadratic computational complexity ($O(L^2)$ with respect to sequence length $L$) makes long-context processing and high-throughput inference incredibly expensive. Linear recurrent attention models (like Mamba or DeltaNet) offer a solution by compressing history into a fixed-size recurrent state, achieving linear scaling ($O(L)$).
However, compressing a massive context into a finite memory space introduces a major challenge: interference. How do you update this compressed memory with new information without scrambling existing, valid associations?
"Delta-rule" models attempt to solve this by reading the memory at the current key location and subtracting it before writing a new value. Yet, prior state-of-the-art models like Gated DeltaNet and Kimi Delta Attention (KDA) suffered from a shared limitation: they used a single scalar gate to control both how much old content to erase (on the key side) and how much new content to commit (on the value side). Because erasing and writing act on completely different axes of the memory state, tying them to a single scalar acts as a severe modeling bottleneck.
Methods
The authors introduce Gated DeltaNet-2, which decouples the erase and write operations using two separate channel-wise gates:
- Channel-wise Erase Gate ($b_t$): Operates on the key axis, determining exactly which coordinates of the old memory should be cleared.
- Channel-wise Write Gate ($w_t$): Operates on the value axis, deciding which coordinates of the incoming value should be committed to the state.
Mathematically, the Gated Delta Rule-2 update is formulated as: $$S_t = (I - k_t(b_t \odot k_t)^\top) D_t S_{t-1} + k_t(w_t \odot v_t)^\top$$ where $D_t$ represents the channel-wise decay matrix inherited from KDA.
To ensure this model can be trained efficiently on modern hardware, the authors derived a chunkwise WY algorithm. By absorbing the cumulative channel-wise decay into asymmetric rank-one erase factors, they formulated a gate-aware parallel backward pass. This allows the model to utilize dense matrix multiplications inside fixed-size chunks, mapping perfectly to GPU tensor cores via custom Triton kernels.
The authors evaluate Gated DeltaNet-2 in two structural flavors: a recurrent-only model and a hybrid model that pairs the recurrent layer with Sliding-Window Attention (SWA) to handle exact local interactions.
Datasets
The models were pretrained from scratch on 100B tokens from the FineWeb-Edu dataset.
Evaluation was performed across a diverse suite of standard benchmarks:
- Language Modeling: WikiText and LAMBADA (perplexity and zero-shot accuracy).
- Commonsense Reasoning: A 9-task suite including PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, OpenBookQA, Social IQa, and BoolQ.
- Synthetic Long-Context Retrieval: The RULER benchmark, specifically testing Single Needle-In-A-Haystack (S-NIAH) and Multi-Key Needle-In-A-Haystack (MK-NIAH) tasks up to 8K context lengths.
- Real-World Retrieval: A recall-heavy suite spanning SWDE (HTML extraction), FDA (PDF key-value retrieval), SQuAD, TriviaQA, Natural Questions (NQ), and DROP.
Key Insights and Ideas
- The Asymmetry of Memory Editing: Erasing is fundamentally a key-side operation (identifying where to clear), while writing is a value-side operation (identifying what features to store). Decoupling them allows the model to clear broad contexts via decay, wipe out specific stale associations via erase, and isolate specific value channels to persist via write.
- The Erase Gate is the MVP: Through gate structure ablations, the authors discovered that the channel-wise erase gate ($b_t$) contributes the majority of the performance gains compared to the write gate ($w_t$). Selectively protecting or revising key channels is more critical for preventing interference than reweighting written values.
- Hardware Efficiency Preserved: Introducing complex, channel-wise gating mechanisms often breaks the parallelizability of RNNs. Gated DeltaNet-2 bypasses this by sharing the same lower-triangular WY inverse matrix across both the erase-side and write-side auxiliary computations, keeping training fast.
Key Results
At a matched 1.3B parameter scale, Gated DeltaNet-2 achieved the strongest overall results against Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants.
- Long-Context Mastery: On the interference-heavy Multi-Key Needle-In-A-Haystack task (MK-NIAH-1) at a 4K context length, the recurrent Gated DeltaNet-2 scored 37.8%, vastly outperforming KDA (28.0%) and Mamba-2 (21.4%).
- Real-World Retrieval: In hybrid configurations, Gated DeltaNet-2 achieved the highest average accuracy (42.28%) across real-world tasks like SQuAD and TriviaQA.
- Flat Throughput Scaling: As shown in Figure 2, Gated DeltaNet-2 maintains a highly efficient, near-flat training throughput profile on an H100 GPU as sequence length scales from 2K to 16K, whereas standard Transformers suffer a sharp performance degradation.
Limitations and Future Work
- Erase Range Constraints: The authors experimented with expanding the erase gate range from $[0, 1]^{d_k}$ to $[0, 2]^{d_k}$ to allow for over-erasing (negative eigenvalues), but found it provided no consistent gains at the 1.3B parameter scale.
- Scale Limitations: The evaluation is bounded to a 1.3B parameter model trained on 100B tokens. Testing how well these decoupled gates scale to larger parameter sizes (e.g., 7B+) and trillion-token regimes remains an area for future exploration.
Summary updated