Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models

Dmytro Babarytskyi, Oleksandr BabarytskyiarXiv 2026 · arXiv

TL;DR

This paper introduces the Dynamic Wave layer—a recurrent sequence mixer that uses damped oscillations to maintain memory—and demonstrates that a hybrid architecture (interleaving Wave and Attention layers) outperforms both pure Transformers and pure recurrent models in language modeling quality. Crucially, the authors provide a "reality check" on efficiency, showing that the massive memory gains often claimed for sub-quadratic models largely disappear when compared against modern, fused attention implementations.

What Problem Does This Paper Address?

Standard Large Language Models (LLMs) rely on Multi-Head Attention (MHA), which has a "quadratic bottleneck": the computational cost and memory required grow with the square of the sequence length ($O(N^2)$). While researchers have proposed sub-quadratic alternatives like State-Space Models (SSMs) and linear attention, these often struggle with "associative recall"—the ability to pinpoint specific information from deep within a long prompt.

Furthermore, the authors identify a methodological gap: many papers claim recurrent layers are "more efficient" without specifying their baseline. If you compare a new layer to a naive, "unfused" attention implementation, it looks revolutionary. If you compare it to the highly optimized "fused" kernels (like FlashAttention) used in production, the advantage might vanish.

Methods

The authors formalize the Dynamic Wave layer, which treats each hidden state channel as a driven, damped harmonic oscillator.

Fig. 2. Computation graph of the Dynamic Wave layer at step t.

Unlike standard RNNs that just decay, the Wave layer uses three input-dependent parameters:

  1. $\alpha_t$ (Forgetting rate): Controls exponential damping.
  2. $\omega_t$ (Frequency): Controls the oscillation speed.
  3. $\phi_t$ (Phase): Controls the starting point of the wave.

This allows the model to selectively reinforce or cancel out information over time. The authors also prove this recurrence is an associative scan, meaning it can be parallelized during training just like Mamba or S4.

They evaluate three 10.94M-parameter configurations:

  • Pure Transformer: Four blocks of standard MHA.
  • Pure Wave: Four blocks of the Dynamic Wave layer.
  • Hybrid Wave-Attention: Two Wave blocks followed by two Attention blocks.
Fig. 1. Layer configurations of the three evaluated 10.94M-parameter models.

Datasets

The models were trained and evaluated on a 500,000-story subset of TinyStories, a dataset of syntactically simple children's stories designed to help small models learn coherent English. Evaluation was performed on a held-out split of 2,000 chunks not seen during training.

Key Insights and Ideas

  • Hybridization is a Quality Play: The Hybrid model achieved the best perplexity. The authors suggest a "division of labor": lower Wave layers perform cheap, sequential compression of local context, while upper Attention layers perform global associative retrieval on that compressed representation.
  • The "Efficiency" Illusion: Against a naive attention baseline, the Pure Wave layer uses ~79% less memory at 4096 tokens. However, against a fused scaled-dot-product attention kernel, the memory advantage is essentially zero. In fact, during training, the Wave layer was 8.3% more expensive because it has to store more intermediate states for backpropagation.
  • Implementation > Architecture: Most of the "recurrent latency penalty" isn't due to the sequential nature of the math, but rather unoptimized code. By simply "hoisting" element-wise operations (like the exp and cos functions) out of the sequential loop, the authors sped up the Wave layer by 2.53x without writing a single line of custom CUDA code.
  • The Vocabulary Shadow: In smaller models, the memory used by the final "logit" layer (the vocabulary projection) is so massive that it masks the memory differences between mixers until you reach very long context lengths (roughly >833 tokens in their setup).

Key Results

  • Quality: The Hybrid model reached a perplexity of 5.95, significantly better than the Pure Transformer (6.93) and Pure Wave (7.26).
  • Interpretability: Attention maps show the Hybrid model's attention layers focusing on specific content words (like "tree" or "Tom"), while the Wave layers' damping coefficients ($\alpha_t$) spike at the end of sentences to "flush" the memory state.

Attention map of the first Attention layer in the Hybrid model. Dynamics of the memory damping coefficient αt in the first Wave layer of the Hybrid model.

  • Memory: At a 4096-token context, Pure Wave and Pure Transformer (fused) both used ~1404 MB of VRAM. The Hybrid model also matched this figure, inheriting the cost of its most expensive component.

Limitations and Future Work

  • Scale: The study was limited to 11M-parameter models. It remains to be seen if the Hybrid quality advantage holds at the billion-parameter scale.
  • Inference Throughput: The current implementation lacks a "state-caching" path for inference, causing throughput to collapse in unbounded contexts. Future work needs to implement an incremental decoding interface.
  • Benchmarking: The authors suggest future comparisons should include established sub-quadratic baselines like Mamba or RWKV at matched parameter budgets.
  • Hardware: All profiling was done in FP32 on a single Tesla T4 GPU; results may vary with mixed-precision (FP16/BF16) or multi-GPU setups.

Summary updated

Abstract

Large Language Models (LLMs) rely on Multi-Head Attention (MHA) for sequence modeling, but scaling to long contexts is constrained by the quadratic cost of attention, motivating sub-quadratic recurrent alternatives. This paper formalizes a Dynamic Wave recurrent layer, based on input-dependent exponential damping and cosine phase modulation, and evaluates three 10.94M-parameter configurations (Pure Transformer, Pure Wave, and Hybrid Wave-Attention) on TinyStories under perplexity, latency, throughput and peak VRAM profiling up to 4096 tokens. The Hybrid configuration reaches the best perplexity, 5.95 on the full held-out split against 6.93 for the Transformer and 7.26 for Pure Wave. The efficiency picture, however, is governed by implementation rather than by architecture. Against an unfused attention baseline, Pure Wave uses 78.7% less peak VRAM at a 4096-token context (1404.3 MB vs. 6601.6 MB); against a fused scaled-dot-product attention kernel the same comparison yields no advantage at all (1404.3 MB vs. 1403.6 MB), and in training mode Pure Wave is 8.3% more expensive than the Transformer. Likewise, hoisting elementwise terms out of the recurrence loop speeds the layer up 2.53-fold with no custom kernel, reducing its latency gap to attention from 2.48-fold to 1.12-fold. A hybrid stack is found to inherit the cost of its more expensive mixer rather than the advantage of its cheaper one, its peak VRAM matching the Pure Transformer to within 0.1%. A closed-form memory model reproduces these measurements and predicts the context length beyond which attention overtakes the vocabulary projection. The findings indicate that reported efficiency gains for recurrent attention replacements must be stated relative to a named attention backend and a named implementation, and that hybridization is best justified on quality grounds.
BibTeX
@misc{babarytskyi2026attention,
  author = {Dmytro Babarytskyi and Oleksandr Babarytskyi},
  title = {{Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models}},
  year = {2026},
  eprint = {2610.39854},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39854},
}
Figure