Don’t stop pretraining! Efficiently building specialised language models in resource-constrained settings.

Sven Najem-Meyer, Frédéric Kaplan, Matteo RomanelloNAACL 2025

TL;DR

This paper investigates how to best build specialized language models for domains with limited data, like classical scholarship. They found that adapting existing large models (like XLM-R) is consistently more effective than retraining from scratch, even with small adaptation datasets. Surprisingly, domain-specific tokenizers didn't always improve performance, and token density emerged as a more reliable indicator of model success.

What Problem Does This Paper Address?

Building high-performing language models for specialized domains or rare languages is a significant challenge. Typically, researchers face a choice: either adapt a general-purpose pre-trained model by continuing its training on domain-specific data (adaptation), or train a new model from scratch using only domain-specific data (retraining). While adaptation preserves the vast linguistic knowledge learned during initial pre-training, it can be hampered by a generic model's vocabulary, which might be ill-suited for specialized terms or languages. Retraining offers the flexibility to create a custom tokenizer tailored to the domain, but it risks losing the benefits of large-scale pre-training, especially when domain-specific data is scarce. This paper aims to resolve this trade-off by systematically evaluating the impact of tokenization strategies and specialization approaches (adaptation vs. retraining) in the context of classical scholarship, a field characterized by multilingualism, code-switching, and unique vocabulary.

Methods

The researchers employed a multi-faceted evaluation approach, combining intrinsic and extrinsic methods.

  • Intrinsic Evaluation: They used "fertility," a metric measuring how many tokens are needed to represent a word, to assess tokenizers. Lower fertility indicates a more efficient representation.
  • Extrinsic Evaluation: Models were evaluated on four downstream tasks within classical scholarship: Latin part-of-speech tagging, bibliographical entity recognition, multilingual named entity recognition, and text anchor recognition. Macro-average F1 score, precision, and recall were used as evaluation metrics.
  • Model Architectures: They experimented with two primary multilingual transformer encoders:
  • XLM-ROBERTa-base (XLM-R): A subword-based model with a large, generic SentencePiece tokenizer.
  • CANINE-C: A character-based model with a broad Unicode point vocabulary.
  • Specialization Strategies: They compared:
  • Retraining: Training models from scratch on domain-specific data.
  • Adapting: Further pre-training existing generic models on domain-specific data.
  • Tokenization: They investigated the impact of:
  • Generic Tokenizers: Using the original tokenizer of a pre-trained model.
  • Domain-Specific Tokenizers: Training new tokenizers on the domain corpus.
  • Character-Based Tokenizers: Models that operate directly on characters.

Datasets

The primary dataset used for training and evaluation was the Classical Scholarship Corpus (CSC). This newly assembled corpus contains 1.4 billion tokens across six languages (ancient Greek, Latin, English, French, German, and Italian) and is described as the largest corpus of its kind for classical scholarship.

Specific downstream task datasets included:

  • EvaLatin: For Latin part-of-speech tagging.
  • EpiBau: For bibliographical entity recognition.
  • AjMC-NE-Corpus: For multilingual named entity recognition.
  • AjMC-LL-Corpus: For text anchor recognition.

Key Insights and Ideas

  • Adaptation Trumps Retraining: The most significant finding is that adapting existing large language models consistently outperforms retraining from scratch, even when the adaptation data is limited. This highlights the immense value of the knowledge embedded in large pre-trained models.
  • Token Density Over Fertility: Contrary to expectations, intrinsic metrics like fertility (lower is better) did not reliably predict downstream performance. Instead, "token density" – the frequency of tokens in the corpus – emerged as a more crucial factor. Models with denser token distributions (meaning fewer unique tokens are used more frequently) performed better, especially in resource-constrained settings.
  • Domain-Specific Tokenizers Aren't Always Better: The study found that in-domain tokenizers did not necessarily lead to improved performance. This challenges a common assumption and suggests that the benefits of domain-specific tokenization might be less pronounced than previously thought, or that other factors (like token density) are more dominant.
  • Character-Based Models Show Adaptability: While generally underperforming subword models like XLM-R, character-based models like CANINE-C demonstrated significant gains when adapted to the domain. This suggests they could be a viable alternative for highly specialized or underrepresented languages where subword tokenization is particularly problematic.
  • Small Adaptation Corpora Can Be Powerful: Even with relatively small adaptation datasets (e.g., 300 million tokens), significant performance gains can be achieved, reinforcing the efficiency of the adaptation strategy.

Key Results

  • Adaptation Superiority: Adapted XLM-R models consistently outperformed retrained and generic models across all downstream tasks. For instance, XLM-RAD (adapted) achieved the best overall performance.
  • Generic Tokenizer Advantage: Surprisingly, the model retrained with the generic XLM-R tokenizer (XLM-RRT-G-250) outperformed models retrained with domain-specific tokenizers, showing an 8.4% improvement in F1 score.
  • Character Model Adaptation Gains: Adapting CANINE-C led to improvements of up to 15% in F1 score for certain tasks, significantly narrowing the performance gap with XLM-R.
  • Data Ablation Impact: Retraining models on significantly reduced datasets led to a dramatic performance drop (22.9% F1 score), whereas adaptation showed much less sensitivity to data reduction.

Limitations and Future Work

  • Model Architecture Differences: The comparison between CANINE-C and XLM-R is somewhat limited by architectural differences and distinct pre-training objectives, making it hard to isolate the impact of character-based tokenization alone.
  • Tokenizer Training Corpus Size: The authors hypothesize that the limited size of the corpus used to train the domain-specific tokenizers might have hindered their development, suggesting that larger tokenization training corpora could yield better results.
  • Future Work: The paper suggests further research into the potential of character-based models and exploring adaptation as a pathway for specializing large language models, especially in scenarios where retraining is infeasible. They also propose investigating these findings in other historical domains.

Summary updated

BibTeX
@inproceedings{najemmeyer2025dont,
  author = {Sven Najem-Meyer and Frédéric Kaplan and Matteo Romanello},
  title = {{Don’t stop pretraining! Efficiently building specialised language models in resource-constrained settings.}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure