Birdie: Advancing State Space Models with Reward-Driven Objectives and Curricula

Sam Blouir, Jimmy T. H. Smith, Antonios Anastasopoulos, Amarda ShehuarXiv 2024 · DOI · arXiv

Abstract

State Space Models (SSMs) offer a promising alternative to Transformers due to their linear scaling with sequence length. However, they traditionally struggle with precise data retrieval and long-range reasoning. This paper introduces Birdie, a training framework that enhances SSMs through bidirectional context processing and a reinforcement-learning-driven curriculum of specialized objectives. Our results demonstrate that Birdie-trained SSMs can rival Transformers in complex retrieval tasks without sacrificing computational efficiency.

Introduction

The current landscape of large language models is dominated by the trade-off between the high accuracy of Transformers and the efficiency of State Space Models (SSMs). While Transformers excel at capturing complex dependencies, their quadratic scaling makes them prohibitively expensive for very long sequences. Conversely, SSMs like Mamba and Hawk scale linearly but suffer from a "memory bottleneck," often failing to retrieve specific information from their compressed hidden states.

Birdie addresses this limitation not by altering the fundamental architecture of SSMs, but by optimizing the training process. By introducing a dynamic curriculum of retrieval-focused tasks, Birdie teaches SSMs to utilize their memory more effectively.

Methodology

The Birdie framework incorporates three core innovations:

1. Bidirectional Context Processing

Standard SSMs are typically causal, processing information only from left to right. Birdie introduces a bidirectional encoder for the initial prompt. By partitioning the model's internal state into forward and reverse components, the model can "look ahead" during the encoding phase, allowing for a more organized representation of the context before generation begins.

2. Specialized Pre-training Objectives

Moving beyond simple next-token prediction, Birdie utilizes a mixture of denoising tasks designed to stress-test and improve the model's retrieval capabilities. These tasks include selective copying, sentence deshuffling, and span infilling.

Table 1: Overview of specialized pre-training objectives used in the Birdie framework.

3. Dynamic RL Curriculum

To manage the complexity of multiple training objectives, Birdie employs a Multi-Armed Bandit (MAB) framework. This reinforcement learning agent acts as a teacher, adaptively adjusting the mixture of tasks based on the model's real-time performance. This ensures that the model focuses on the most informative tasks at each stage of training, avoiding the suboptimality of fixed-ratio mixtures.

Experimental Results

We evaluated Birdie-trained models on a variety of benchmarks, focusing on retrieval-intensive tasks.

Retrieval Stress Tests

In the Multi-Phone Number Retrieval task, models were required to find specific numbers within a 16,000-token context. Standard SSMs trained with next-token prediction showed a rapid decline in accuracy as the number of items to retrieve increased. In contrast, Birdie-trained SSMs maintained high accuracy, significantly narrowing the performance gap with Transformers.

Figure 1: Comparison of digit retrieval accuracy across different model architectures and training procedures.

General Benchmarks

Birdie-trained models also showed substantial improvements on standard reading comprehension tasks. For instance, a Birdie-trained Hawk model achieved 55.8% accuracy on SQuAD V2, a dramatic increase from the 16.6% achieved by the baseline version. Importantly, these gains in retrieval did not come at the expense of general knowledge; Birdie models maintained competitive performance on benchmarks like MMLU.

Conclusion

Birdie demonstrates that the perceived architectural weaknesses of SSMs in data retrieval can be largely mitigated through sophisticated training strategies. By combining bidirectional processing with an adaptive, reward-driven curriculum, Birdie enables efficient SSMs to perform at levels previously reserved for more computationally expensive Transformer models. Future work will explore the scalability of this approach to larger model sizes and even more extensive datasets.

Summary updated

Abstract

Efficient state space models (SSMs), such as linear recurrent neural networks and linear attention variants, offer computational advantages over Transformers but struggle with tasks requiring long-range in-context retrieval-like text copying, associative recall, and question answering over long contexts. Previous efforts to address these challenges have focused on architectural modifications, often reintroducing computational inefficiencies. In this paper, we propose a novel training procedure, Birdie, that significantly enhances the in-context retrieval capabilities of SSMs without altering their architecture. Our approach combines bidirectional input processing with dynamic mixtures of specialized pre-training objectives, optimized via reinforcement learning. We introduce a new bidirectional SSM architecture that seamlessly transitions from bidirectional context processing to causal generation. Experimental evaluations demonstrate that Birdie markedly improves performance on retrieval-intensive tasks such as multi-number phone book lookup, long paragraph question-answering, and infilling. This narrows the performance gap with Transformers, while retaining computational efficiency. Our findings highlight the importance of training procedures in leveraging the fixed-state capacity of SSMs, offering a new direction to advance their capabilities. All code and pre-trained models are available at https://www.github.com/samblouir/birdie, with support for JAX and PyTorch.
BibTeX
@misc{blouir2024birdie,
  author = {Sam Blouir and Jimmy T. H. Smith and Antonios Anastasopoulos and Amarda Shehu},
  title = {{Birdie: Advancing State Space Models with Reward-Driven Objectives and Curricula}},
  year = {2024},
  eprint = {2411.01030},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.01030},
}
Figure