Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences

Neehal Tumma, Noel Loo, Daniela Rus

Introduction

As Large Language Models (LLMs) scale to process increasingly long contexts, the quadratic computational cost of standard softmax attention has become a significant bottleneck. To address this, researchers have developed subquadratic recurrent models like Mamba-2 and DeltaNet. Under the Test-Time Regression (TTR) framework, these models can be viewed as solving an online least-squares problem during their forward pass, learning a linear map from "keys" to "values."

However, existing delta-rule recurrences act as first-order gradient descent steps, ignoring the "curvature" of the data. This means they fail to account for the fact that some directions in the feature space are stretched more than others, resulting in a coarse approximation of the ideal solution. While exact second-order solvers like MesaNet exist, they are often numerically unstable or difficult to parallelize on modern GPUs. This paper introduces a middle ground: Preconditioned DeltaNet (PDN), a method that captures second-order curvature information while remaining fast, stable, and highly parallelizable.

The Preconditioned DeltaNet Approach

The authors introduce PDN and its variants, Preconditioned Gated DeltaNet (PGDN) and Preconditioned Kimi Delta Attention (PKDA), built on four core pillars:

  1. Theoretical Equivalence: The authors prove that if an exact preconditioner (the inverse key Gram matrix) is used, Linear Attention (an "offline" solver) and DeltaNet (an "online" solver) become mathematically equivalent.
  2. Diagonal Approximation: Because computing a full inverse matrix is computationally prohibitive ($O(d^3)$), the authors propose a diagonal approximation. This approximation tracks the coordinate-wise second moments of the keys, allowing the model to scale the "write" update for each feature dimension independently.
  3. Stable Parameterization: To prevent numerical instability during training, the authors avoid using the raw inverse. Instead, they utilize a log-space parameterization and a "squash" function that constrains the preconditioner values to a safe range (e.g., $[0.66, 1.5]$). This ensures internal state stability while still leveraging curvature information.
  4. Chunkwise Parallelization: The authors derive new chunkwise parallel algorithms to compute these preconditioned updates efficiently on GPUs, fully utilizing high-speed tensor cores.

Key Insights and Taxonomy

The paper introduces the DGPS (Decay, Gain, Preconditioner, Solve) taxonomy, positing that preconditioning is a distinct architectural lever orthogonal to decay factors (like in Mamba) or gain terms.

The preconditioner effectively acts as a per-dimension learning rate. If a feature is frequently observed, the preconditioner dampens new updates to prevent overwriting stable memories. The authors also distinguish between applying preconditioning to the Write Key (Apply-to-Key, or ATK) versus the Query (Apply-to-Query, or ATQ), finding that ATK is empirically superior for delta-rule models.

Ultimately, the success of PDN lies in its stability: the squashed diagonal version provides the vast majority of the benefits of a theoretically perfect Sherman-Morrison update without the associated deep learning instability.

Empirical Results

The preconditioned models were evaluated across synthetic recall tasks (MQAR) and large-scale language modeling at the 340M and 1B parameter scales (trained on 15 billion and 50 billion tokens of the SlimPajama dataset, respectively).

PDN, PGDN, and PKDA consistently outperformed their unpreconditioned counterparts across almost all benchmarks. Notably, at the 340M scale, PGDN improved average in-context retrieval scores from 24.77 to 26.17. On the RULER long-context benchmark (specifically Single Needle-In-A-Haystack), PGDN maintained significantly higher accuracy at 4K and 8K context lengths compared to standard GDN.

Crucially, these performance gains come with minimal computational cost. The preconditioned kernels incur only a ~10% overhead in training throughput compared to standard DeltaNet and are 20% faster than MesaNet, proving that approximate preconditioning is a highly effective trade-off.

Limitations and Future Work

While highly effective, the diagonal approximation cannot capture correlations between different feature dimensions. Additionally, recent research suggests that allowing negative eigenvalues aids in state tracking; the current preconditioned parameterization does not naturally learn these, presenting an opportunity for future hybrid designs. Finally, while this work focused on preconditioning the "write" side, exploring simultaneous key and query preconditioning could further close the performance gap with full softmax attention.

Summary updated

Abstract

To address the increasing long-context compute limitations of softmax attention, several subquadratic recurrent operators have been developed. This work includes models such as Mamba-2, DeltaNet, Gated DeltaNet (GDN), and Kimi Delta Attention (KDA). As the space of recurrences grows, a parallel line of work has arisen to taxonomize them. One compelling view is the test-time regression (TTR) framework, which interprets recurrences as performing online least squares updates that learn a linear map from the keys to values. Existing delta-rule recurrences can be seen as first-order approximations to this objective, but notably ignore the curvature of the least-squares loss during optimization. In this work, we address this by introducing preconditioning to these recurrences. Starting from the theory of online least squares, we derive equivalences between linear attention and the delta rule in the exactly preconditioned case. Next, we realize this theory in practice by proposing a diagonal approximation: this enables us to introduce preconditioned variants of DeltaNet, GDN, and KDA alongside efficient chunkwise parallel algorithms for computing them. Empirically, we find that our preconditioned delta-rule recurrences yield consistent performance improvements across synthetic recall benchmarks and language modeling at the 340M and 1B scale.
BibTeX
@inproceedings{tumma2026preconditioned,
  author = {Neehal Tumma and Noel Loo and Daniela Rus},
  title = {{Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=UC6YiTOeKb},
}
Figure