How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation

Xin Lu, Yanyan Zhao, Si Wei, Shijin Wang, Bing Qin, Ting Liu

Introduction

As Large Language Models (LLMs) continue to scale, the quadratic computational complexity of the Transformer’s self-attention mechanism has become a significant bottleneck. This has spurred the development of "stateful" architectures—such as Mamba, RWKV, and Gated DeltaNet—which utilize linear recurrence or linear attention to achieve superior efficiency.

While these models often match Transformers in standard perplexity scores, they frequently struggle with specific tasks involving long-context retrieval or complex reasoning. This paper investigates why these efficient architectures often suffer from a hidden degradation in "base capabilities" and identifies the key design principles required to bridge the gap with Transformers.

The Evaluation Gap: Mixed Domain vs. Limited Domain

The authors argue that the current methodology for evaluating new architectures is flawed. Standard "mixed domain" pre-training—where a model is trained on a diverse dataset including code, math, and web text—can mask architectural weaknesses. In such settings, models can "cheat" by learning in-distribution patterns, making it difficult to discern if the architecture itself is capable of general reasoning.

To address this, the researchers proposed a Limited Domain Pre-training stress test. Models are trained on a restricted subset of data (e.g., web crawls from CommonCrawl and C4) and then evaluated on Out-of-Distribution (OOD) data, such as GitHub code or ArXiv papers. This zero-shot transfer approach highlights how well an architecture generalizes to new logic and structures, exposing the fundamental limitations of stateful models.

The Three Pillars of Base Capability

Through systematic ablation and analysis of models like Mamba, the researchers identified three critical factors that prevent the degradation of base capabilities:

  1. Full-Sequence Visibility: The model must be able to "see" the entire history of the sequence, rather than relying on a compressed summary.
  2. Real Relation Calculation: The architecture requires a query-key interaction to explicitly determine the relevance of past information.
  3. Non-Uniform Distribution: The model must be capable of focusing intensely on specific, sparse information rather than averaging or smoothing over the history.

The core finding is that arbitrary selection—the ability to reach back into the sequence and retrieve a specific piece of information—is more important for model capability than complex compression schemes used in RNN-like models.

Proposed Architecture: Top-1 Chunk Selection

Based on these insights, the authors designed a minimalist proof-of-concept architecture called Top-1 Chunk Selection. Unlike standard Transformers that attend to every token, this model identifies and selects only the single most relevant chunk of tokens for any given query.

By utilizing optimized GPU kernels, this approach maintains the "selection" capability of Transformers while achieving the speed of linear models. It ensures that the model can always access the most pertinent historical data without the information loss inherent in fixed-size state compression.

Experimental Results

OOD Performance and Generalization

In the limited domain tests, standard stateful models (Mamba-1, Mamba-2, and RWKV-7) showed significant performance drops when moving from web text to OOD data like code. In contrast, the Top-1 Chunk Selection model matched the Transformer's OOD performance, proving that the selection mechanism is the primary driver of generalization.

Efficiency and Scaling

The Top-1 Chunk Selection model demonstrated significant efficiency gains:

  • Pre-training Speed: Achieved a 4.5x speedup compared to standard Transformers.
  • Inference Speed: Achieved a 1.2x speedup in inference, reaching efficiency levels comparable to Mamba-2.

Long-Context Retrieval

On the "Needle In A Haystack" (S-NIAH) test, which requires retrieving a specific fact from a large body of text, the Top-1 Chunk model maintained high accuracy across context lengths of up to 100k tokens. Other stateful models saw their performance collapse as the sequence length grew, highlighting their struggle with long-range dependencies.

Conclusion

This research reveals that the perceived trade-off between efficiency and capability in sequence modeling is not inevitable. By prioritizing full-sequence arbitrary selection over compression, it is possible to design architectures that are as capable as Transformers while remaining computationally efficient. Future work will explore how these principles scale to trillion-parameter models and whether "Top-K" selection (retrieving multiple chunks) can further enhance performance without sacrificing speed.

Summary updated

BibTeX
@inproceedings{lu2025how,
  author = {Xin Lu and Yanyan Zhao and Si Wei and Shijin Wang and Bing Qin and Ting Liu},
  title = {{How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=vMkJWaa02n},
}
Figure