Introduction
As Large Language Models (LLMs) scale to process increasingly long contexts, the quadratic computational cost of standard softmax attention has become a significant bottleneck. To address this, researchers have developed subquadratic recurrent models like Mamba-2 and DeltaNet. Under the Test-Time Regression (TTR) framework, these models can be viewed as solving an online least-squares problem during their forward pass, learning a linear map from "keys" to "values."
However, existing delta-rule recurrences act as first-order gradient descent steps, ignoring the "curvature" of the data. This means they fail to account for the fact that some directions in the feature space are stretched more than others, resulting in a coarse approximation of the ideal solution. While exact second-order solvers like MesaNet exist, they are often numerically unstable or difficult to parallelize on modern GPUs. This paper introduces a middle ground: Preconditioned DeltaNet (PDN), a method that captures second-order curvature information while remaining fast, stable, and highly parallelizable.
The Preconditioned DeltaNet Approach
The authors introduce PDN and its variants, Preconditioned Gated DeltaNet (PGDN) and Preconditioned Kimi Delta Attention (PKDA), built on four core pillars:
- Theoretical Equivalence: The authors prove that if an exact preconditioner (the inverse key Gram matrix) is used, Linear Attention (an "offline" solver) and DeltaNet (an "online" solver) become mathematically equivalent.
- Diagonal Approximation: Because computing a full inverse matrix is computationally prohibitive ($O(d^3)$), the authors propose a diagonal approximation. This approximation tracks the coordinate-wise second moments of the keys, allowing the model to scale the "write" update for each feature dimension independently.
- Stable Parameterization: To prevent numerical instability during training, the authors avoid using the raw inverse. Instead, they utilize a log-space parameterization and a "squash" function that constrains the preconditioner values to a safe range (e.g., $[0.66, 1.5]$). This ensures internal state stability while still leveraging curvature information.
- Chunkwise Parallelization: The authors derive new chunkwise parallel algorithms to compute these preconditioned updates efficiently on GPUs, fully utilizing high-speed tensor cores.
Key Insights and Taxonomy
The paper introduces the DGPS (Decay, Gain, Preconditioner, Solve) taxonomy, positing that preconditioning is a distinct architectural lever orthogonal to decay factors (like in Mamba) or gain terms.
The preconditioner effectively acts as a per-dimension learning rate. If a feature is frequently observed, the preconditioner dampens new updates to prevent overwriting stable memories. The authors also distinguish between applying preconditioning to the Write Key (Apply-to-Key, or ATK) versus the Query (Apply-to-Query, or ATQ), finding that ATK is empirically superior for delta-rule models.
Ultimately, the success of PDN lies in its stability: the squashed diagonal version provides the vast majority of the benefits of a theoretically perfect Sherman-Morrison update without the associated deep learning instability.
Empirical Results
The preconditioned models were evaluated across synthetic recall tasks (MQAR) and large-scale language modeling at the 340M and 1B parameter scales (trained on 15 billion and 50 billion tokens of the SlimPajama dataset, respectively).
PDN, PGDN, and PKDA consistently outperformed their unpreconditioned counterparts across almost all benchmarks. Notably, at the 340M scale, PGDN improved average in-context retrieval scores from 24.77 to 26.17. On the RULER long-context benchmark (specifically Single Needle-In-A-Haystack), PGDN maintained significantly higher accuracy at 4K and 8K context lengths compared to standard GDN.
Crucially, these performance gains come with minimal computational cost. The preconditioned kernels incur only a ~10% overhead in training throughput compared to standard DeltaNet and are 20% faster than MesaNet, proving that approximate preconditioning is a highly effective trade-off.
Limitations and Future Work
While highly effective, the diagonal approximation cannot capture correlations between different feature dimensions. Additionally, recent research suggests that allowing negative eigenvalues aids in state tracking; the current preconditioned parameterization does not naturally learn these, presenting an opportunity for future hybrid designs. Finally, while this work focused on preconditioning the "write" side, exploring simultaneous key and query preconditioning could further close the performance gap with full softmax attention.
Summary updated