Introduction
Evaluating long-sequence models by training them from scratch on benchmarks like the Long Range Arena (LRA) artificially inflates the performance gap between novel architectures—like State Space Models (SSMs)—and standard Transformers.
Modeling long-range dependencies in sequences is notoriously difficult. Standard Transformers struggle due to their quadratic complexity and lack of inductive biases for long distances. Consequently, a wave of novel architectures, most notably SSMs like S4, have been developed, showing massive performance gains over Transformers on stress-test benchmarks like LRA.
However, there is a disconnect: in the real world, Transformers are almost always pretrained (e.g., LLMs), but on benchmarks like LRA, models are evaluated by training them from scratch (random initialization) using only supervised labels. The authors argue that training from scratch ignores the powerful inductive biases learned through self-supervised pretraining. This creates an "apples-to-oranges" comparison, leading the community to grossly underestimate the inherent capabilities of standard architectures and overestimate the necessity of complex, manually engineered architectural tweaks.
The Solution: Self-Pretraining (SPT)
To level the playing field without introducing external data confounders, the authors propose Self-Pretraining (SPT).
Instead of pretraining on massive external datasets (which would make it hard to isolate architectural capabilities), they pretrain the models only on the downstream task's training data. They use standard self-supervised denoising objectives:
- Causal sequence modeling (next-token prediction) for unidirectional models.
- Masked sequence modeling (predicting hidden tokens, like BERT) for bidirectional models.
After this self-supervised pretraining phase, the models are fine-tuned on the actual supervised task labels. This approach acts as a data-driven initialization method, allowing a fair comparison between architectures since only the task data is used.
Key Results: Closing the Gap
When given the chance to learn data-driven priors via SPT, vanilla Transformers are highly capable of modeling long sequences. The prevailing narrative that standard attention fails at long-range dependencies is largely an artifact of training from scratch.
- Massive Transformer Boost: Self-pretraining improves the average test accuracy of vanilla Transformers on LRA by roughly 30% compared to training from scratch (jumping from 53.60% to 86.21%).
- Transformers Match S4: Properly pretrained bidirectional Transformers achieve an average LRA score of 86.21%, effectively matching the performance of a well-tuned S4 model (86.09% trained from scratch, 86.75% pretrained).
- SOTA on PathX-256: S4 combined with SPT solves the notoriously difficult PathX-256 task (sequence length 65K), jumping 20 absolute points from 67.82% to 87.11%.
Key Insights: Simpler Models and Redundant Priors
SSMs like S4 rely on complex, mathematically derived initializations (like HiPPO) to force the model to remember long sequences. The authors demonstrate that SPT learns these priors directly from the data, rendering manual initialization largely unnecessary.
Because SPT provides such strong data-driven priors, much simpler architectures can match the performance of highly complex, heavily engineered SSMs.
- DLR Matches S4: A simple Diagonal Linear RNN (DLR) with SPT scores 83.37% on a subset of LRA, beating a trained-from-scratch S4 (82.85%) and nearly matching an SPT S4 (84.47%).
- Data-driven kernels adapt to modality: Visualizing the learned convolution kernels in S4 shows that SPT allows the model to adapt its decay rates to the specific data structure (e.g., aligning perfectly with the 2D grid in PathX). Standard S4 uses rigid, data-agnostic decay rates that cannot adapt this way.
- Low-data regimes benefit most: The relative performance gains of SPT over training from scratch are most pronounced when training data is scarce, highlighting its utility as a powerful regularizer.
Limitations and Future Work
Self-pretraining inherently requires more compute than simply training from scratch. While the authors demonstrate that SPT still outperforms training from scratch in a strictly compute-tied setting (where total epochs are fixed), it remains an extra step in the pipeline.
Future work should focus on reducing the computational requirements of this pretraining evaluation scheme. Ultimately, the authors strongly advocate that the machine learning community adopt pretraining as a mandatory step when benchmarking new architectures to ensure reliable, fair performance estimation.
Summary updated