SLAY: Geometry-Aware Spherical Linearized Attention with Yat-Kernel

Jose Miguel Luna, Taha Bouhsine, Krzysztof Choromański

Standard Transformers are the powerhouses of modern AI, but they have a "quadratic bottleneck." As the input sequence length ($L$) grows, the computational cost of the attention mechanism explodes at a rate of $O(L^2)$. This makes processing long documents or high-resolution images prohibitively expensive.

A new paper, "SLAY: Geometry-Aware Spherical Linearized Attention with Yat-Kernel," introduces a clever way to break this bottleneck. By borrowing concepts from physics and using advanced mathematical linearization, the authors propose an attention mechanism that scales linearly ($O(L)$) while maintaining the high performance of standard Transformers.

TL;DR

SLAY (Spherical Linearized Attention with Yat-Kernels) is a new linear-time attention mechanism that uses a physics-inspired "Yat-kernel" to achieve $O(L)$ complexity. It enables Transformers to handle sequences up to 30x longer than standard models while delivering performance that is nearly indistinguishable from traditional softmax attention.

What Problem Does This Paper Address?

The core of a Transformer is Softmax Attention. While effective, it has two major flaws:

  1. Quadratic Scaling: It requires calculating a similarity score for every pair of tokens in a sequence, which is impossible for very long contexts.
  2. Magnitude Conflation: Softmax similarity depends on both the alignment (angle) and the magnitude of the query and key vectors. This can lead to unbounded growth and requires careful normalization to keep training stable.

Previous "Linear Attention" methods (like Performers or Cosformers) tried to solve the scaling issue by approximating the softmax kernel. However, these often suffer from "approximation error" or numerical instability (like negative attention scores), leading to a significant drop in model quality compared to the original.

Methods

The authors introduce SLAY, which replaces softmax with a geometry-aware kernel called the Yat-kernel (or $\mathbb{E}$-Product).

Key technical steps include:

  • Spherical Constraint: SLAY forces all query and key vectors onto a unit sphere. This ensures that attention depends purely on the angular alignment between tokens, making the geometry much more predictable.
  • The Yat-Kernel: Inspired by inverse-square interactions in physics, this kernel naturally penalizes distant vectors and rewards alignment.
  • Bernstein’s Theorem & Quadrature: To make the kernel linear-time, the authors use Bernstein’s theorem to rewrite the complex Yat-kernel as an integral of simpler "polynomial-exponential" parts. They then use Gauss-Laguerre quadrature to turn this integral into a finite sum of features.
  • Positive Random Features (PRFs): To ensure the model never crashes due to "negative attention" (a common fail state in linear models), they use specific random feature mappings that are guaranteed to be strictly positive.

Datasets

The researchers put SLAY through a rigorous battery of tests:

  • Synthetic Benchmarks: 22 different tasks designed to test core capabilities like counting, sorting, long-range memory, and reasoning.
  • Extreme Classification: The Eurlex-4K dataset, which tests the model's ability to handle a massive number of labels.
  • Language Modeling: They trained a GPT-2 Small (124M parameters) architecture from scratch on 2.5 billion tokens, following the Chinchilla scaling laws to ensure a fair comparison with standard models.

Key Insights and Ideas

  • Physics as a Blueprint: The Yat-kernel mimics how physical forces (like gravity or electromagnetism) interact. This "inverse-square" logic provides a natural "self-regularizing" effect that softmax lacks.
  • Selectivity is Key: The authors found that SLAY is more "selective" than softmax. It peaks sharply for highly aligned tokens and drops to near-zero for others, whereas softmax often "blurs" the distinction between relevant and irrelevant tokens.
  • Stability through Positivity: By ensuring every component of their approximation is non-negative, SLAY avoids the "denominator problem" (where the normalization term in linear attention accidentally hits zero or goes negative), making it much more robust during training.

Key Results

  • Scaling Power: SLAY successfully processed sequences up to 131K tokens on a single GPU, whereas standard softmax attention ran out of memory (OOM) at just 16K tokens.
  • Near-Parity with Softmax: In full-scale language modeling, SLAY achieved a validation loss of 4.6760, remarkably close to standard softmax's 4.6417.
  • Outperforming Peers: SLAY consistently beat other linear-time methods like Performers (Val Loss: 5.4524) and Cosformers (Val Loss: 5.1983) by a wide margin.
  • Efficiency: It provides a 30x increase in sequence length capability without sacrificing the "representational strength" that makes Transformers so good at language.

Limitations and Future Work

While SLAY is a major step forward, the authors note a few areas for further exploration:

  • Spherical Normalization: The requirement to project queries and keys onto a unit sphere is a specific architectural choice that might need tuning when integrated into existing pre-trained models.
  • Approximation Overhead: While $O(L)$, the constant factor (the number of random features needed) can be high. Future work could look into optimizing the "sketching" techniques used to fuse these features.
  • Scaling Up: The current results are on a GPT-2 Small scale. Testing SLAY on multi-billion parameter models will be the ultimate test of its "near-softmax" performance claims.

Summary updated

Abstract

We propose a new class of linear-time attention mechanisms based on a relaxed and computationally efficient formulation of the recently introduced E-Product, often referred to as the Yat-kernel (Bouhsine, 2025). The resulting interactions are geometry-aware and inspired by inverse-square interactions in physics. Our method, Spherical Linearized Attention with Yat Kernels (SLAY), constrains queries and keys to the unit sphere so that attention depends only on angular alignment. Using Bernstein's theorem, we express the spherical Yat-kernel as a nonnegative mixture of polynomial-exponential product kernels and derive a strictly positive random-feature approximation enabling linear-time O(L) attention. We establish positive definiteness and boundedness on the sphere and show that the estimator yields well-defined, nonnegative attention scores. Empirically, SLAY achieves performance that is nearly indistinguishable from standard softmax attention while retaining linear time and memory scaling, and consistently outperforms prior linear-time attention mechanisms such as Performers and Cosformers. To the best of our knowledge, SLAY represents the closest linear-time approximation to softmax attention reported to date, enabling scalable Transformers without the typical performance trade-offs of attention linearization.
BibTeX
@misc{luna2026slay,
  author = {Jose Miguel Luna and Taha Bouhsine and Krzysztof Choromański},
  title = {{SLAY: Geometry-Aware Spherical Linearized Attention with Yat-Kernel}},
  year = {2026},
  eprint = {2602.04915},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.04915},
}
Figure