Abstract
State Space Models (SSMs) provide a computationally efficient alternative to Transformers for long-sequence modeling, yet they often struggle with retrieval-intensive tasks due to their fixed-size internal state. This paper introduces Birdie, a novel training framework that addresses this "retrieval bottleneck" through bidirectional prefix processing and a dynamic mixture of training objectives. By employing a reinforcement learning-driven curriculum, Birdie enables SSMs to effectively utilize their state capacity, closing the performance gap with Transformers on long-context challenges while maintaining linear scaling efficiency.
Introduction
While Transformers dominate modern AI, their memory requirements (the KV cache) grow linearly with sequence length, making them prohibitively expensive for long-document processing. State Space Models (SSMs) like Mamba or Hawk offer a promising alternative with fixed-size states and linear scaling. However, SSMs typically suffer from a "retrieval bottleneck," struggling to recall specific details from long contexts—a task essential for applications like legal analysis or large-scale data lookup.
This research argues that the retrieval problem is not solely an architectural flaw but a consequence of standard training procedures. Causal Language Modeling (CLM), or next-word prediction, does not sufficiently incentivize SSMs to use their limited state capacity for efficient retrieval. Birdie solves this by reimagining how these models are trained.
The Birdie Framework
Birdie introduces three fundamental shifts to the SSM training pipeline:
- Bidirectional Prefix Processing: Unlike standard left-to-right SSMs, Birdie allows the model to process the entire input prompt (the prefix) bidirectionally. This enables the model to "triage" information, deciding which context is critical to compress into its fixed state before generation begins.
- Diverse Training Objectives: Birdie moves beyond simple next-token prediction by incorporating a suite of tasks designed to stress-test retrieval and compression capabilities. These include Full Span Corruption (FSC), Deshuffling, and Selective Copying.
- Dynamic RL Mixture: Rather than using a static ratio of tasks, Birdie utilizes a Reinforcement Learning (RL) critic. This multi-armed bandit framework monitors model progress in real-time, adaptively sampling the most effective mixture of objectives to maximize learning efficiency.
Experimental Setup
Models were pre-trained on The Pile (32B tokens) and instruction-tuned using a robust combination of datasets including FLAN, SlimOrca-Dedup, and Zephyr. Evaluation focused on standard benchmarks (EleutherAI LM Harness) alongside specialized long-context tasks:
- Phone Book Lookup: A synthetic task involving up to 800 entries and 16,384 tokens to test dense retrieval.
- SQuAD V2: Long-paragraph question answering.
- Story Infilling: A new benchmark requiring models to fill blanks in multi-entry stories based on context retrieval.
Results and Discussion
Birdie-trained models demonstrated a significant leap in retrieval performance. In the 16k-context phone book task, the Birdie-trained Gated SSM+ achieved 42.0% accuracy, outperforming the CLM-trained version (40.0%) and drastically narrowing the gap with Transformers.
Key findings include:
- Robustness to Context: While CLM-trained SSMs often see performance degrade as context length increases, Birdie-trained models maintain stability comparable to Transformers.
- Efficiency Gains: Birdie-trained SSMs remain significantly faster than their Transformer counterparts. In testing, the Birdie model processed 26,148 tokens/sec/A100, compared to just 8,506 tokens/sec/A100 for a standard Transformer.
- Generalization: The focus on retrieval did not come at the cost of general knowledge; Birdie models performed at or above the level of CLM models on standard language benchmarks.
Conclusion
Birdie demonstrates that the "retrieval bottleneck" in State Space Models can be largely mitigated through superior training objectives and bidirectional processing. By teaching models to compress and recall information more effectively, Birdie allows SSMs to compete with Transformers on complex tasks while retaining their inherent scaling advantages. Future work will explore scaling this framework beyond the 1.4B parameter regime to determine if these benefits persist in larger models.
Summary updated