Standard Transformers face a fundamental "quadratic wall": as the input sequence grows, the cost of self-attention increases with the square of the sequence length. To process a million tokens, a model would traditionally need to compare every token to every other token, leading to prohibitive memory and compute requirements.
Researchers have converged on a powerful solution: Hybrid Architectures. Instead of using global self-attention for everything, these models bifurcate the task. They use high-resolution sliding window attention for immediate local context and an efficient compressed memory (often via linear recurrence or sparse retrieval) for the distant past.
The Local-Global Split
The intuition behind hybrid models is that not all history is equally important. Most linguistic and structural dependencies are local—a word's meaning is usually determined by its immediate neighbors. However, long-range reasoning requires "recalling" specific facts from thousands of tokens back.
Longformer by Ivan Sekulić et al. was an early proponent of this, combining local windowed attention with a few "global" tokens that attend to the entire sequence. More recently, models like RecurrentGemma (Aleksandar Botev et al.) and SpeechSSM (Se Jin Park et al.) formalize this as the "Griffin" architecture, interleaving linear recurrences with local sliding-window attention.
In these designs:
- Sliding Window Attention maintains a high-definition view of the most recent tokens (e.g., the last 2,048 tokens). This preserves the "associative recall" precision that makes Transformers so effective.
- Linear Recurrence or State-Space Models (SSMs) act as a "lossy" but constant-memory summary of everything that has fallen out of the sliding window.
Recurrence as a Compressed Cache
The bottleneck in long-context inference is the Key-Value (KV) cache. Hybrid models like RATTENTION (Ruoming Pang et al.) and LESS (Harry Dong et al.) treat the recurrent state as a "compressed" version of the KV cache.
When a token leaves the sliding window, instead of being deleted, its information is folded into a fixed-size recurrent state. Simran Arora and colleagues demonstrated in BASED that even a tiny sliding window, when paired with linear attention, can recover nearly all the performance of a full Transformer. This "best of both worlds" approach allows for 24x higher throughput because the model no longer needs to store and re-scan millions of raw keys and values.
Training for Long-Term Memory
A surprising discovery in this field is that "less is more" during training. Loïc Cabannes and the Meta FAIR team found in SWAX that using shorter attention windows during training actually forces the recurrent layers to work harder. If the model can't rely on the "crutch" of easy local attention, it develops a much more robust long-term memory in its recurrent states.
Similarly, RecurFormer (Ruiqing Yan et al.) shows that we don't even need hybrid layers; we can have hybrid heads. By identifying which attention heads naturally focus on local patterns, they replace those heads with Mamba (linear RNN) blocks, slashing memory usage by 90% without losing the global "look-back" capability of the remaining attention heads.
Scaling to Millions of Tokens
For truly massive contexts, some hybrids move beyond simple recurrence into sparse retrieval. InfLLM (Chaojun Xiao et al.) uses a sliding window for the "now" but treats the distant past as an external memory bank. Instead of attending to everything, it performs a sparse lookup to grab only the most relevant blocks of history.
By combining these techniques—local precision via windows, global summaries via recurrence, and targeted retrieval via sparsity—models can now process sequences that were previously unthinkable. As Jiaqi Leng et al. showed in their work on Hierarchical Sparse Attention, these principles allow a model trained on just 4,000 tokens to extrapolate to 32 million tokens, effectively solving the length generalization problem through architectural synergy.