TL;DR
This paper introduces the Dynamic Wave layer—a recurrent sequence mixer that uses damped oscillations to maintain memory—and demonstrates that a hybrid architecture (interleaving Wave and Attention layers) outperforms both pure Transformers and pure recurrent models in language modeling quality. Crucially, the authors provide a "reality check" on efficiency, showing that the massive memory gains often claimed for sub-quadratic models largely disappear when compared against modern, fused attention implementations.
What Problem Does This Paper Address?
Standard Large Language Models (LLMs) rely on Multi-Head Attention (MHA), which has a "quadratic bottleneck": the computational cost and memory required grow with the square of the sequence length ($O(N^2)$). While researchers have proposed sub-quadratic alternatives like State-Space Models (SSMs) and linear attention, these often struggle with "associative recall"—the ability to pinpoint specific information from deep within a long prompt.
Furthermore, the authors identify a methodological gap: many papers claim recurrent layers are "more efficient" without specifying their baseline. If you compare a new layer to a naive, "unfused" attention implementation, it looks revolutionary. If you compare it to the highly optimized "fused" kernels (like FlashAttention) used in production, the advantage might vanish.
Methods
The authors formalize the Dynamic Wave layer, which treats each hidden state channel as a driven, damped harmonic oscillator.
Unlike standard RNNs that just decay, the Wave layer uses three input-dependent parameters:
- $\alpha_t$ (Forgetting rate): Controls exponential damping.
- $\omega_t$ (Frequency): Controls the oscillation speed.
- $\phi_t$ (Phase): Controls the starting point of the wave.
This allows the model to selectively reinforce or cancel out information over time. The authors also prove this recurrence is an associative scan, meaning it can be parallelized during training just like Mamba or S4.
They evaluate three 10.94M-parameter configurations:
- Pure Transformer: Four blocks of standard MHA.
- Pure Wave: Four blocks of the Dynamic Wave layer.
- Hybrid Wave-Attention: Two Wave blocks followed by two Attention blocks.
Datasets
The models were trained and evaluated on a 500,000-story subset of TinyStories, a dataset of syntactically simple children's stories designed to help small models learn coherent English. Evaluation was performed on a held-out split of 2,000 chunks not seen during training.
Key Insights and Ideas
- Hybridization is a Quality Play: The Hybrid model achieved the best perplexity. The authors suggest a "division of labor": lower Wave layers perform cheap, sequential compression of local context, while upper Attention layers perform global associative retrieval on that compressed representation.
- The "Efficiency" Illusion: Against a naive attention baseline, the Pure Wave layer uses ~79% less memory at 4096 tokens. However, against a fused scaled-dot-product attention kernel, the memory advantage is essentially zero. In fact, during training, the Wave layer was 8.3% more expensive because it has to store more intermediate states for backpropagation.
- Implementation > Architecture: Most of the "recurrent latency penalty" isn't due to the sequential nature of the math, but rather unoptimized code. By simply "hoisting" element-wise operations (like the
expandcosfunctions) out of the sequential loop, the authors sped up the Wave layer by 2.53x without writing a single line of custom CUDA code. - The Vocabulary Shadow: In smaller models, the memory used by the final "logit" layer (the vocabulary projection) is so massive that it masks the memory differences between mixers until you reach very long context lengths (roughly >833 tokens in their setup).
Key Results
- Quality: The Hybrid model reached a perplexity of 5.95, significantly better than the Pure Transformer (6.93) and Pure Wave (7.26).
- Interpretability: Attention maps show the Hybrid model's attention layers focusing on specific content words (like "tree" or "Tom"), while the Wave layers' damping coefficients ($\alpha_t$) spike at the end of sentences to "flush" the memory state.

- Memory: At a 4096-token context, Pure Wave and Pure Transformer (fused) both used ~1404 MB of VRAM. The Hybrid model also matched this figure, inheriting the cost of its most expensive component.
Limitations and Future Work
- Scale: The study was limited to 11M-parameter models. It remains to be seen if the Hybrid quality advantage holds at the billion-parameter scale.
- Inference Throughput: The current implementation lacks a "state-caching" path for inference, causing throughput to collapse in unbounded contexts. Future work needs to implement an incremental decoding interface.
- Benchmarking: The authors suggest future comparisons should include established sub-quadratic baselines like Mamba or RWKV at matched parameter budgets.
- Hardware: All profiling was done in FP32 on a single Tesla T4 GPU; results may vary with mixed-precision (FP16/BF16) or multi-GPU setups.
Summary updated