Abstract
Modeling long-range dependencies across sequences is a longstanding goal in machine learning and has led to architectures, such as state space models, that dramatically outperform Transformers on long sequences. However, these impressive empirical gains have been by and large demonstrated on benchmarks (e.g. Long Range Arena), where models are randomly initialized and trained to predict a target label from an input sequence. In this work, we show that random initialization leads to gross overestimation of the differences between architectures and that pretraining with standard denoising objectives, using only the downstream task data, leads to dramatic gains across multiple architectures and to very small gaps between Transformers and state space models (SSMs). In stark contrast to prior works, we find vanilla Transformers to match the performance of S4 on Long Range Arena when properly pretrained, and we improve the best reported results of SSMs on the PathX-256 task by 20 absolute points. Subsequently, we analyze the utility of previously-proposed structured parameterizations for SSMs and show they become mostly redundant in the presence of data-driven initialization obtained through pretraining. Our work shows that, when evaluating different architectures on supervised tasks, incorporation of data-driven priors via pretraining is essential for reliable performance estimation, and can be done efficiently.
Introduction
The machine learning community has long observed a discrepancy between the success of Transformers in large-scale foundation models and their perceived failure on long-range dependency benchmarks like the Long Range Arena (LRA). This perceived weakness spurred the development of specialized architectures, notably State Space Models (SSMs) like S4, which utilize complex mathematical priors to handle long sequences.
This paper argues that the "Transformer gap" is largely an artifact of benchmarking methodology. While foundation models benefit from massive pretraining, benchmark evaluations typically rely on training from scratch with random initialization. This forces models to learn fundamental data structures and the target task simultaneously, often disadvantaging architectures without specific hard-coded inductive biases.
Methodology: Self-Pretraining (SPT)
The authors propose Self-Pretraining (SPT) as a fairer evaluation protocol. SPT utilizes the existing training data of a benchmark task for a self-supervised "denoising" phase before supervised fine-tuning. Crucially, no external data is introduced.
- Bidirectional models (e.g., S4, bidirectional Transformers) are pretrained using Masked Language Modeling.
- Unidirectional models are pretrained using Causal/Autoregressive modeling.
This approach provides a "warm start" by allowing the model to learn data-driven priors directly from the task's own distribution.
Experimental Results
The primary evaluation utilizes the LRA benchmark, including tasks like ListOps, Text, Image classification, and the challenging PathX-256 (sequence length 65,536).
Closing the Architectural Gap
When applying SPT, the performance gap between vanilla Transformers and specialized SSMs effectively vanishes. As shown in Figure 1, Transformers with SPT achieve a mean LRA performance of ~86%, matching S4. Furthermore, S4 with SPT reaches 87.11% on PathX-256, a 20-point improvement over previous state-of-the-art results.
Redundancy of Manual Inductive Biases
A key finding is that the complex, structured parameterizations often cited as necessary for SSM success become redundant when data-driven priors are available. Simple Diagonal Linear RNNs (DLR), which typically struggle when trained from scratch, match the performance of the more complex S4 model once both are pretrained.
Discussion and Insights
- Learned Kernels: Analysis of convolution kernels reveals that SPT allows models to learn task-specific structures, such as the 2D grid layout of flattened images, which random initialization fails to capture.
- Data Efficiency: The relative benefits of SPT are most pronounced when labeled data is scarce, as the pretraining phase extracts maximum utility from the available unlabeled input sequences.
- Scientific Rigor: The authors conclude that "training from scratch" is an unreliable proxy for architectural potential. While SPT adds computational overhead, it is necessary for a scientifically sound comparison of long-sequence models.
Conclusion
By introducing Self-Pretraining, this work demonstrates that many perceived architectural advantages in long-sequence modeling are actually compensations for poor initialization. When given the benefit of data-driven priors, vanilla Transformers remain highly competitive, and the complexity of specialized models can be significantly reduced.
Summary updated