On the importance of pre-training data volume for compact language models

Vincent Micheli, Martin d'Hoffschmidt, François Fleuret

Introduction

The prevailing paradigm in Natural Language Processing (NLP) has long been "bigger is better," leading to an arms race of massive models like GPT-3 and RoBERTa trained on hundreds of gigabytes of text. While these models achieve state-of-the-art results, they impose significant barriers: the computational resources required are prohibitive for most researchers, and many languages or specialized domains lack the necessary data volume. This paper challenges this dogma by investigating the performance of compact language models trained on significantly reduced datasets.

The CamemBERT_SMALL Architecture

To explore the efficiency of smaller models, the researchers introduced CamemBERT_SMALL, a compact version of the French CamemBERT model. While it retains a 12-layer structure, it utilizes a much smaller hidden dimension of 256 and only 4 attention heads. This results in a model with just 17 million parameters—approximately 6.5 times smaller than the standard BERT-base.

The primary advantage of this architecture is its operational efficiency. As shown in the table below, CamemBERT_SMALL is significantly faster and lighter than its larger counterparts, making it suitable for deployment on resource-constrained devices.

Comparison of model size and inference time across CamemBERT variants.

Methodology and Datasets

The study employed the French portion of the OSCAR corpus, a massive multilingual dataset. The researchers created six overlapping subsets of increasing size, ranging from 10 MB to 4,000 MB (4 GB), to pre-train multiple versions of CamemBERT_SMALL using the Masked Language Modeling (MLM) objective.

Evaluation was conducted using the French Question Answering Dataset (FQuAD), which contains 60,000 question-answer pairs based on Wikipedia articles. This allowed the team to measure how pre-training data volume directly impacts performance on a complex downstream task.

Key Results: The 100 MB Sweet Spot

The most significant finding of the study is the rapid plateauing of performance as data volume increases. While model "perplexity"—a measure of how well the model predicts masked tokens—continues to improve with more data, the actual utility for downstream tasks like Question Answering levels off very early.

Performance metrics of CamemBERT_SMALL across varying pre-training data volumes.

As illustrated in the results, a model trained on only 100 MB of text achieved an F1 score of 68.50. Increasing the training data by 40 times to 4,000 MB only yielded a marginal improvement to an F1 score of 69.76. This suggests that for compact architectures, model capacity becomes a bottleneck long before data scarcity does.

Insights and Implications

The research highlights several critical takeaways for sustainable AI:

  • Diminishing Returns: For small models, adding massive amounts of data provides negligible benefits once a certain threshold (around 100 MB) is reached.
  • Pre-training Necessity: Even minimal pre-training (10 MB) is vastly superior to no pre-training, but the "extra" step of domain-specific MLM training often provides little added value once a baseline of pre-training is established.
  • Accessibility: These findings suggest that high-quality NLP tools can be developed for low-resource languages or specialized domains using a fraction of the data and compute previously thought necessary.

Limitations and Future Work

While promising, the study focused primarily on Question Answering. More complex tasks might still benefit from larger datasets or longer training durations. Additionally, the researchers kept the number of training steps fixed at 200k; it remains to be seen if larger datasets would eventually pull ahead if trained for millions of steps. Future research could also investigate if these findings apply to other compression techniques such as pruning or knowledge distillation.

Summary updated

Abstract

Recent advances in language modeling have led to computationally intensive and resource-demanding state-of-the-art models. In an effort towards sustainable practices, we study the impact of pre-training data volume on compact language models. Multiple BERT-based models are trained on gradually increasing amounts of French text. Through fine-tuning on the French Question Answering Dataset (FQuAD), we observe that well-performing models are obtained with as little as 100 MB of text. In addition, we show that past critically low amounts of pre-training data, an intermediate pre-training step on the task-specific corpus does not yield substantial improvements.
BibTeX
@inproceedings{micheli2020importance,
  author = {Vincent Micheli and Martin d'Hoffschmidt and François Fleuret},
  title = {{On the importance of pre-training data volume for compact language models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=AAmNEcxGjZR2},
}
Figure