TL;DR
MoM (Mixture-of-Memories) is a novel architecture that brings the "Mixture-of-Experts" philosophy to the hidden states of linear sequence models. By routing tokens to multiple independent memory states instead of compressing everything into one, it significantly boosts memory capacity and reduces data interference. This allows linear models to finally rival Transformers on recall-heavy tasks without losing their efficiency edge.
What Problem Does This Paper Address?
Modern AI is caught in a tug-of-war between computational efficiency and "memory" capacity.
Transformers are the gold standard for performance because they keep a "KV cache"—essentially a perfect record of every token seen so far. This makes them amazing at recall-intensive tasks (like finding a specific fact in a long document), but it also makes them slow and memory-hungry as sequences grow, suffering from $O(n^2)$ complexity.
Linear Sequence Models (like SSMs, Linear Attention, and RNNs) were designed to fix this. They compress the entire history into a single, fixed-size "memory state." This makes them incredibly fast ($O(n)$ training, $O(1)$ inference), but it creates a bottleneck. When new information comes in, it often overwrites or "interferes" with old information. This "memory interference" leads to poor performance on tasks that require precise long-term recall.
The authors of MoM ask: Can we have the high capacity of a Transformer with the linear efficiency of an RNN?
Methods
The core idea of MoM is to replace the single, overloaded memory state of a linear model with a Mixture-of-Memories.
- The Router: For every incoming token, a lightweight router network calculates "importance scores" and selects the top-$k$ most relevant memory states (out of $N$ total).
- Independent Memory States: Instead of one big matrix, MoM maintains multiple independent RNN-like states. Only the selected $k$ memories are updated by the current token, which prevents the other memories from being "polluted" by irrelevant information.
- Shared Memory: To ensure the model doesn't lose the "big picture," MoM includes a continuously updated shared memory that sees every token.
- Memory Mixing: The final output is a weighted sum of the activated memories, queried by the current token.
- Hardware-Efficient Implementation: To avoid the computational overhead of managing many small memories, the authors developed a "varlen" (variable length) approach. They reorder tokens into buckets based on their assigned memory and process them in parallel using a custom Triton kernel.
Crucially, MoM is a general framework. It can be used with almost any linear update rule, including Gated DeltaNet, GLA, or RetNet.
Datasets
The researchers evaluated MoM using a rigorous suite of benchmarks:
- Pre-training: Trained from scratch on SlimPajama (15B tokens for the 380M model; 100B tokens for the 1.3B model).
- Recall-Intensive Tasks: FDA, SWDE, SQuAD, NQ, TriviaQA, and Drop.
- Long Context: LongBench, which tests summarization, few-shot learning, and synthetic "needle-in-a-haystack" tasks.
- Common-sense Reasoning: ARC (Easy/Challenge), HellaSwag, Lambada, PIQA, and WinoGrande.
Key Insights and Ideas
- Memory Specialization: Using UMAP visualizations, the authors found that the router isn't just picking memories at random. It learns to specialize: different memory states focus on distinct "clusters" of information in the feature space.
- TTT Ensemble Learning: The authors frame MoM as a form of "Test-Time Training (TTT) Ensemble." Because each memory only has to fit a simpler sub-distribution of the data, the learning difficulty for the model is reduced compared to trying to cram everything into one state.
- Separation > Gating: Most linear models try to solve interference using "forget gates" (deciding what to delete). MoM argues that spatial separation (storing different things in different places) is a much more powerful way to preserve information.
Key Results
- Closing the Gap: At the 1.3B parameter scale, MoM achieved an average score of 36.04 on recall tasks, coming remarkably close to the 37.31 of a standard Transformer++. For context, other strong linear models like GLA (31.93) and RetNet (26.45) lagged far behind.
- Efficiency at Scale: MoM maintains linear complexity. In benchmarks, it successfully processed 512k sequence lengths where Transformers hit Out-of-Memory (OOM) errors.
- Long Context Mastery: On the LongBench benchmark, MoM outperformed all other linear baselines, particularly in code completion and synthetic tasks that require high-fidelity memory.
- Hybrid Potential: The authors found that "hybridizing" the model—inserting just three Transformer layers into a 24-layer MoM stack—pushed performance even higher, surpassing the pure Transformer baseline.
Limitations and Future Work
- Parameter Count vs. Compute: While MoM is computationally efficient (sparse activation), it does increase the total number of parameters in the model. The authors argue this is a fair trade-off for the performance gains, but it may impact disk storage and VRAM requirements for the weights.
- Routing Strategy: The current model uses a simple top-$k$ router. Future work could explore more sophisticated routing mechanisms or "expert" architectures specifically tuned for memory rather than just feed-forward layers.
- Scaling Laws: While the 1.3B results are promising, it remains to be seen how MoM scales to massive 70B+ parameter models where routing stability becomes a larger engineering challenge.
Summary updated