TL;DR
Linear sequence models (like SSMs and linear RNNs) are highly efficient but struggle with recall-intensive tasks because they compress entire histories into a single fixed-size memory state. To fix this, Mixture-of-Memories (MoM) introduces a routing network that dynamically distributes tokens across multiple independent memory states. This bio-inspired approach drastically increases memory capacity and reduces interference, matching Transformer performance while maintaining linear training and constant inference complexity.
What Problem Does This Paper Address?
While Transformers remain the gold standard for sequence modeling due to their virtually unlimited memory capacity (via the KV cache), their quadratic time complexity $O(n^2)$ makes scaling to long sequences computationally brutal.
To solve this, researchers developed linear sequence models—such as linear attention, State Space Models (SSMs like Mamba), and linear RNNs—which achieve $O(n)$ training and $O(1)$ inference complexity. However, they pay a steep price: they compress the entire past sequence into a single, fixed-size hidden state. This extreme compression introduces two major flaws: limited memory capacity and memory interference (where new information overwrites and degrades old representations). Consequently, linear models perform poorly on recall-intensive tasks where retrieving specific, fine-grained details from long contexts is critical.
Methods
Inspired by how the human hippocampus uses theta-gamma neural oscillations to segregate memory items and prevent interference, the authors propose Mixture-of-Memories (MoM).
Instead of a single memory state, MoM maintains multiple independent recurrent memory states embedded within a linear recurrent mechanism:
- Routing Mechanism: For each input token, a simple linear router calculates importance scores. A softmax function then selects the
top-kmemory states to activate and update. Unactivated memories remain untouched, completely avoiding interference from irrelevant tokens. - Shared Memory: A continuously active shared memory runs in parallel to capture global, long-term dependencies across the entire sequence.
- Memory Mixing: The final output is generated by performing a weighted sum of the activated memory states based on the router's scores, which is then queried by the token's query vector.
MoM is highly flexible and acts as a plug-and-play framework compatible with almost any underlying linear model's update rule (e.g., GLA, Mamba2, DeltaNet, RWKV7).
Hardware-Efficient Implementation
To prevent the slow, fragmented computations typically caused by sparse routing, the authors developed a custom Triton kernel. It reorders tokens based on routing results, groups them into a variable-length (varlen) sequence for parallel processing, and then restores the original sequence order.
Datasets
- Pre-training: Models were trained from scratch on the SlimPajama dataset (15B tokens for the 380M parameter model; 100B tokens for the 1.3B parameter model).
- Evaluation Benchmarks:
- Recall-Intensive Tasks: FDA, SWDE, SQuAD, Natural Questions (NQ), TriviaQA, and Drop.
- Long Context: The LongBench benchmark (encompassing single/multi-doc QA, summarization, few-shot learning, synthetic tasks, and code completion).
- Common-Sense Reasoning: WikiText, LAMBADA, ARC-easy, ARC-challenge, HellaSwag, PiQA, and WinoGrande.
- Length Extrapolation: Evaluated using the FineWeb dataset.
Key Insights and Ideas
- Separation Beats Expansion: Simply scaling up the size of a single RNN state (expanding the hidden dimension) is less effective than splitting the memory into distinct, independent pools. Forcing multifaceted sequence information into one shared space inherently causes mutual interference.
- Spontaneous Specialization: Analysis of the router showed that memories naturally specialize without explicit prompting. For instance, Memory 1 naturally handles basic nouns/verbs/prepositions (semantic/causal logic), while Memory 2 focuses on proper nouns and scientific terms (specialized knowledge).
- Hybridization Potential: Integrating just a few global attention layers (1 Transformer layer after every 7 MoM layers) yields massive performance boosts, outperforming pure Transformers on recall tasks while remaining highly efficient.
Key Results
- Closing the Transformer Gap: On recall-intensive tasks at the 1.3B parameter scale, MoM achieved an average score of 36.04%, significantly outperforming its baseline Gated DeltaNet (32.30%) and coming remarkably close to Transformer++ (37.31%).
- Long Context Superiority: In LongBench evaluations, MoM achieved an average score of 15.64, beating all linear baselines (e.g., Gated DeltaNet at 13.98, GSA at 14.61).
- True Linear Efficiency: MoM demonstrates strict linear complexity for inference latency and memory consumption, completely avoiding the Out-Of-Memory (OOM) errors that plague Transformers at lengths like 512k tokens.
- Excellent Extrapolation: When trained on 2k sequence lengths, MoM maintained stable, low perplexity when extrapolated up to 32k context lengths, whereas Transformers experienced severe perplexity degradation past their training threshold.
Limitations and Future Work
- Parameter Overhead: Applying sparse activation to the key and value projections slightly increases the total parameter count of the model, though the number of activated parameters per token remains controlled.
- Advanced Routing: While an auxiliary loss ensures balanced routing across memories, more sophisticated routing strategies could further enhance memory specialization.
- Token vs. Channel Mixing Hybrids: The authors suggest exploring architectures where MoM handles token mixing (sequence memory) while traditional Mixture-of-Experts (MoE) handles channel mixing (FFN layers) to get the best of both worlds.
Summary updated