MoM: Linear Sequence Modeling with Mixture-of-Memories

Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, Tao Zhang, Yu Cheng

TL;DR

MoM (Mixture-of-Memories) is a novel architecture that brings the "Mixture-of-Experts" philosophy to the hidden states of linear sequence models. By routing tokens to multiple independent memory states instead of compressing everything into one, it significantly boosts memory capacity and reduces data interference. This allows linear models to finally rival Transformers on recall-heavy tasks without losing their efficiency edge.

What Problem Does This Paper Address?

Modern AI is caught in a tug-of-war between computational efficiency and "memory" capacity.

Transformers are the gold standard for performance because they keep a "KV cache"—essentially a perfect record of every token seen so far. This makes them amazing at recall-intensive tasks (like finding a specific fact in a long document), but it also makes them slow and memory-hungry as sequences grow, suffering from $O(n^2)$ complexity.

Linear Sequence Models (like SSMs, Linear Attention, and RNNs) were designed to fix this. They compress the entire history into a single, fixed-size "memory state." This makes them incredibly fast ($O(n)$ training, $O(1)$ inference), but it creates a bottleneck. When new information comes in, it often overwrites or "interferes" with old information. This "memory interference" leads to poor performance on tasks that require precise long-term recall.

The authors of MoM ask: Can we have the high capacity of a Transformer with the linear efficiency of an RNN?

Methods

The core idea of MoM is to replace the single, overloaded memory state of a linear model with a Mixture-of-Memories.

MoM Architecture
  1. The Router: For every incoming token, a lightweight router network calculates "importance scores" and selects the top-$k$ most relevant memory states (out of $N$ total).
  2. Independent Memory States: Instead of one big matrix, MoM maintains multiple independent RNN-like states. Only the selected $k$ memories are updated by the current token, which prevents the other memories from being "polluted" by irrelevant information.
  3. Shared Memory: To ensure the model doesn't lose the "big picture," MoM includes a continuously updated shared memory that sees every token.
  4. Memory Mixing: The final output is a weighted sum of the activated memories, queried by the current token.
  5. Hardware-Efficient Implementation: To avoid the computational overhead of managing many small memories, the authors developed a "varlen" (variable length) approach. They reorder tokens into buckets based on their assigned memory and process them in parallel using a custom Triton kernel.

Crucially, MoM is a general framework. It can be used with almost any linear update rule, including Gated DeltaNet, GLA, or RetNet.

Datasets

The researchers evaluated MoM using a rigorous suite of benchmarks:

  • Pre-training: Trained from scratch on SlimPajama (15B tokens for the 380M model; 100B tokens for the 1.3B model).
  • Recall-Intensive Tasks: FDA, SWDE, SQuAD, NQ, TriviaQA, and Drop.
  • Long Context: LongBench, which tests summarization, few-shot learning, and synthetic "needle-in-a-haystack" tasks.
  • Common-sense Reasoning: ARC (Easy/Challenge), HellaSwag, Lambada, PIQA, and WinoGrande.

Key Insights and Ideas

  • Memory Specialization: Using UMAP visualizations, the authors found that the router isn't just picking memories at random. It learns to specialize: different memory states focus on distinct "clusters" of information in the feature space.
  • TTT Ensemble Learning: The authors frame MoM as a form of "Test-Time Training (TTT) Ensemble." Because each memory only has to fit a simpler sub-distribution of the data, the learning difficulty for the model is reduced compared to trying to cram everything into one state.
  • Separation > Gating: Most linear models try to solve interference using "forget gates" (deciding what to delete). MoM argues that spatial separation (storing different things in different places) is a much more powerful way to preserve information.

Key Results

  • Closing the Gap: At the 1.3B parameter scale, MoM achieved an average score of 36.04 on recall tasks, coming remarkably close to the 37.31 of a standard Transformer++. For context, other strong linear models like GLA (31.93) and RetNet (26.45) lagged far behind.
  • Efficiency at Scale: MoM maintains linear complexity. In benchmarks, it successfully processed 512k sequence lengths where Transformers hit Out-of-Memory (OOM) errors.
Latency and Memory vs Sequence Length
  • Long Context Mastery: On the LongBench benchmark, MoM outperformed all other linear baselines, particularly in code completion and synthetic tasks that require high-fidelity memory.
  • Hybrid Potential: The authors found that "hybridizing" the model—inserting just three Transformer layers into a 24-layer MoM stack—pushed performance even higher, surpassing the pure Transformer baseline.

Limitations and Future Work

  • Parameter Count vs. Compute: While MoM is computationally efficient (sparse activation), it does increase the total number of parameters in the model. The authors argue this is a fair trade-off for the performance gains, but it may impact disk storage and VRAM requirements for the weights.
  • Routing Strategy: The current model uses a simple top-$k$ router. Future work could explore more sophisticated routing mechanisms or "expert" architectures specifically tuned for memory rather than just feed-forward layers.
  • Scaling Laws: While the 1.3B results are promising, it remains to be seen how MoM scales to massive 70B+ parameter models where routing stability becomes a larger engineering challenge.

Summary updated

Abstract

Linear sequence modeling methods, such as linear attention, state space modeling, and linear RNNs, offer significant efficiency improvements by reducing the complexity of training and inference. However, these methods typically compress the entire input sequence into a single fixed-size memory state, which leads to suboptimal performance on recall-intensive tasks. To address this limitation, we introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes multiple independent memory states, with a router network directing input tokens to specific memory states. This approach greatly enhances the overall memory capacity while minimizing memory interference. MoM serves as a general framework that can be seamlessly combined with diverse memory update mechanisms across linear models. As a result, MoM performs exceptionally well on recall-intensive tasks, surpassing existing linear sequence modeling techniques. Despite incorporating multiple memory states, the computation of each memory state remains linear in complexity, allowing MoM to retain the linear-complexity advantage during training, while constant-complexity during inference. Our experimental results show that MoM outperforms current linear sequence models on downstream language tasks, particularly recall-intensive tasks, and even achieves performance comparable to Transformer models.
BibTeX
@inproceedings{du2026mom,
  author = {Jusen Du and Weigao Sun and Disen Lan and Jiaxi Hu and Tao Zhang and Yu Cheng},
  title = {{MoM: Linear Sequence Modeling with Mixture-of-Memories}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=3PdOq8Rgue},
}
Figure