Abstract
This paper is an extended abstract of our ICLR 2024 Outstanding Paper Award work. Modeling long-range dependencies across sequences is a longstanding goal in machine learning. While state space models reportedly outperform Transformers on benchmarks like Long Range Arena, we show that random initialization significantly overestimates architectural differences. Pretraining with standard denoising objectives on downstream task data leads to dramatic gains across architectures and minimal performance gaps between Transformers and state space models (SSMs). We demonstrate that properly pretrained vanilla Transformers match S4 performance on Long Range Arena and improve SSM results on PathX-256 by 20 absolute points. Our analysis shows previously-proposed structured parameterizations for SSMs become largely redundant with pretraining. When evaluating architectures on supervised tasks, incorporating data-driven priors via pretraining is essential for reliable performance estimation.
Introduction
For years, the machine learning community has used the Long Range Arena (LRA) benchmark to argue that standard Transformers are poorly suited for capturing long-range dependencies. Because Transformers performed poorly when trained from scratch on these tasks, researchers developed complex new architectures—most notably State Space Models (SSMs) like S4—which use sophisticated mathematical parameterizations to "force" the model to remember distant information.
This work challenges the notion that these architectural differences are fundamental. We ask: Is the Transformer's failure due to its architecture, or is it simply because training from scratch with random initialization is a "rigged" comparison? In the era of Large Language Models (LLMs), we know Transformers excel after pretraining, yet we continue to evaluate new "long-sequence" architectures by training them from zero on small datasets. This practice significantly overestimates architectural differences by ignoring the role of data-driven priors.
Methods: Self-Pretraining (SPT)
We introduce a simple, inexpensive protocol called Self-Pretraining (SPT). Instead of training a model directly on a supervised task (like classification), we first pretrain the model on the exact same training data using a self-supervised objective. This requires no additional data.
Two variants are explored:
- Masked SPT: Hiding parts of the input and requiring the model to reconstruct them (similar to BERT).
- Causal SPT: Predicting the next token in the sequence (similar to GPT).
After this "warm-up" phase, the model is fine-tuned on the actual supervised task. We apply this to vanilla Transformers and compare them against state-of-the-art SSMs like S4.
Key Results
Our experiments on the LRA benchmark reveal that the performance gap between Transformers and SSMs largely vanishes when both are allowed to learn from the data before being evaluated on the supervised task.
As shown in Figure 1, vanilla Transformers with SPT improve their average LRA performance from 53.60% to 86.21%, effectively matching the performance of S4 (86.09%). Furthermore, SPT provides a massive boost to SSMs on the challenging PathX-256 task (sequence length 65,536), where S4's accuracy jumps from 67.82% to 87.11%.
Detailed results in Table 2 demonstrate that Transformers with Rotary embeddings and Masked SPT achieve competitive results across all LRA sub-tasks, including ListOps, Text, Retrieval, and Image. Notably, the performance of pretrained Transformers is on par with or superior to many specialized architectures designed specifically for long-range dependencies.
Discussion: Inductive Biases vs. Data-Driven Priors
A key insight of this work is that many "hand-crafted" inductive biases in SSMs—such as specialized kernel decay patterns or structured parameterizations—become largely redundant once a model undergoes SPT. The model learns the necessary patterns directly from the data. For instance, we found that with SPT, a simple Diagonal Linear RNN (DLR) can match the performance of the much more mathematically complex S4 model.
While SPT levels the playing field for accuracy, it does not change the fundamental computational complexity of the architectures. SSMs still maintain log-linear complexity regarding sequence length, making them more efficient for inference on extremely long sequences compared to the quadratic cost of standard Transformers. However, for many tasks, the perceived "architectural superiority" of SSMs was actually a reflection of the difficulty of learning from scratch.
Conclusion
This paper argues that "training from scratch" is an unrealistic proxy for model capability. For a fair comparison, every architecture should be allowed to develop data-driven priors through self-supervision before being judged. When properly pretrained, vanilla Transformers are remarkably capable of handling long-range dependencies, suggesting that the community should shift focus from complex architectural constraints to more effective pretraining strategies.
Summary updated