To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks

Sinong Wang, Madian Khabsa, Hao Ma

Abstract

While pretrained models like BERT and RoBERTa are the undisputed leaders in low-resource NLP, their relative advantage diminishes as the volume of downstream training data grows. This paper investigates the "pretraining dogma" by comparing state-of-the-art Transformers against simpler BiLSTM architectures on datasets containing millions of labeled examples. Our findings reveal that for resource-rich text classification, a simple BiLSTM trained from scratch can achieve accuracy within 1% of RoBERTa-Large while offering significantly higher inference speeds and reduced computational overhead.

Introduction

In contemporary NLP, the standard paradigm involves finetuning a massive model pretrained on generic corpora (e.g., RoBERTa) for specific downstream tasks. This approach is highly effective when labeled data is scarce. However, many industrial applications possess millions of labeled data points. This study asks: at what point does the sheer volume of supervised data make expensive pretraining redundant? We explore the trade-off between the marginal accuracy gains of Transformers and the operational efficiency of simpler architectures in resource-rich scenarios.

Experimental Setup

The researchers conducted a head-to-head scaling analysis across three large-scale sentiment classification datasets:

  • Yelp Challenge: 6.69 million reviews.
  • Amazon Sports & Outdoors: 11.9 million reviews.
  • Amazon Electronics: 18.6 million reviews.

Three model categories were evaluated:

  1. RoBERTa (Base and Large): State-of-the-art pretrained Transformers.
  2. Vanilla BiLSTM: Standard bidirectional LSTMs trained from scratch with randomly initialized embeddings.
  3. BiLSTM + Pretrained Embeddings: Hybrid models using frozen RoBERTa token embeddings with a BiLSTM sequence layer.

Results: The Law of Diminishing Returns

The study observed a clear convergence in performance as training data increased. In low-data regimes (e.g., 1% of the dataset), RoBERTa maintains a massive lead, often outperforming LSTMs by nearly 9%. However, as the training set scales into the millions, the "accuracy gap" shrinks dramatically.

Accuracy gap between BiLSTM variants and RoBERTa-Large across three datasets as training data increases.

As shown in the scaling plots, the accuracy gap drops to approximately 1% once the models are exposed to the full breadth of the supervised data. This suggests that the structural advantages of the Transformer architecture and the knowledge gained during pretraining are eventually "leveled" by the sheer volume of task-specific supervision.

Efficiency and Model Complexity

Beyond raw accuracy, the study highlights the practical advantages of simpler models in production environments. While RoBERTa-Large achieves the highest absolute accuracy, the marginal gain over a well-configured BiLSTM is often less than 1% on the largest datasets.

Comparison of accuracy, performance gap (Δ), and parameter counts across models and datasets.

As detailed in the results table, the BiLSTM models are significantly more parameter-efficient. RoBERTa-Large utilizes 304 million parameters, whereas competitive BiLSTM variants range from 2.4 million to 25 million parameters. In terms of operational performance, the BiLSTM models were found to be at least 20x faster at inference on both CPU and GPU hardware.

Key Insights

  • Embeddings are Critical: A significant portion of the pretraining benefit is encapsulated in the token embeddings. Utilizing pretrained embeddings within an LSTM architecture closed much of the performance gap even before reaching maximum data scales.
  • The Production Reality Check: In many industrial contexts, a <1% accuracy improvement does not justify the 20x increase in latency and computational cost associated with deploying large Transformer models.

Conclusion and Limitations

This research demonstrates that for simple semantic tasks like sentiment classification, massive supervised data can effectively compensate for a lack of pretraining. However, the authors note that these findings may vary for more complex tasks requiring deep logical reasoning or long-range dependency modeling, such as natural language inference or complex question answering. Future work will explore whether these convergence patterns hold for more linguistically demanding architectures and tasks.

Summary updated

Abstract

Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks. This paper examines the benefits of pretrained models as a function of the number of training samples used in the downstream task. On several text classification tasks, we show that as the number of training examples grow into the millions, the accuracy gap between finetuning BERT-based model and training vanilla LSTM from scratch narrows to within 1%. Our findings indicate that MLM-based models might reach a diminishing return point as the supervised data size increases significantly.
BibTeX
@inproceedings{wang2020pretrain,
  author = {Sinong Wang and Madian Khabsa and Hao Ma},
  title = {{To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.200},
}
Figure