Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?

Firat Öncel, Matthias Bethge, Beyza Ermis, Mirco Ravanelli, Cem Subakan, Çagatay Yildiz

Abstract

In the last decade, the generalization and adaptation abilities of deep learning models were typically evaluated on fixed training and test distributions. Contrary to traditional deep learning, large language models (LLMs) are (i) even more overparameterized, (ii) trained on unlabeled text corpora curated from the Internet with minimal human intervention, and (iii) trained in an online fashion. These stark contrasts prevent researchers from transferring lessons learned on model generalization and adaptation in deep learning contexts to LLMs. To this end, our short paper introduces empirical observations that aim to shed light on further training of already pretrained language models. Specifically, we demonstrate that training a model on a text domain could degrade its perplexity on the test portion of the same domain. We observe with our subsequent analysis that the performance degradation is positively correlated with the similarity between the additional and the original pretraining dataset of the LLM. Our further token-level perplexity analysis reveals that the perplexity degradation is due to a handful of tokens that are not informative about the domain. We hope these findings will guide us in determining when to adapt a model vs when to rely on its foundational capabilities.

Introduction

In the world of machine learning, the "more is better" philosophy usually reigns supreme: more data and more training cycles typically lead to better performance. However, recent research presented at EMNLP 2024 challenges this assumption for Large Language Models (LLMs). Traditionally, if a practitioner wants a model to perform better on specialized texts (e.g., medical or legal), they perform "continued pretraining" on a domain-specific corpus. The expectation is that the model's perplexity—a measure of its predictive accuracy—will decrease as it adapts.

However, modern LLMs are trained on massive, often undisclosed portions of the internet. This creates a potential "data leakage" problem where the "new" domain might already be well-represented in the model's original training data. This paper investigates whether additional pretraining is always beneficial for models that have already seen a vast, unknown portion of the web.

Methodology

The researchers conducted a systematic study by taking several open-weight models—the GPT-2 family (Small, Large, XLarge), OLMo-1B, and LLaMA-7B—and further training them on specific domains. Each model underwent one epoch of additional pretraining on 20 distinct domains from the Massively Multi-Domain Dataset (M2D2). These domains were categorized into Wikipedia (Wiki), which is generally similar to web-scale training data, and S2ORC (Semantic Scholar), which contains specialized scientific domains less likely to be over-represented in general web crawls.

To understand the drivers of performance change, the authors employed Maximum Mean Discrepancy (MMD) and Fréchet Distance (FD) to compare the embeddings of the adaptation data to samples from the models' original training sets. Furthermore, they performed a granular token-level analysis to identify specific tokens contributing to perplexity changes.

Results and Analysis

The Similarity Paradox

The most striking finding is the "adaptation degradation" phenomenon. While adaptation to S2ORC domains almost always improved performance, adaptation to Wikipedia domains often increased perplexity, making the model worse. As shown in the figure below, there is a strong correlation between domain similarity and performance change. Domains that are highly similar to the original training data (indicated by the similarity metrics MMD and FD) tend to show degradation after additional training.

Performance improvement vs. domain similarity across various LLM architectures.

The "Newline" Culprit and Model Capacity

Token-level analysis revealed that degradation wasn't caused by the model forgetting domain-specific facts. Instead, it was largely driven by a few "uninformative" tokens, particularly structural characters like newlines (\n). The model essentially becomes "confused" by the specific formatting of the adaptation dataset.

Additionally, model size plays a role in resilience. While all models exhibited this phenomenon, larger models like LLaMA-7B showed a smaller gap between zero-shot and adapted performance compared to smaller models like GPT-2 Small, suggesting that increased capacity acts as a buffer against adaptation degradation.

Conclusion

This research highlights a critical challenge in LLM adaptation: more training data is not always better, especially when that data overlaps significantly with the model's prior knowledge. For practitioners, this underscores the importance of knowing a model's training history—a difficult task for closed-source models. Future work should focus on developing better heuristics to decide when to stop adaptation based on the behavior of domain-specific versus general tokens, moving beyond simple perplexity averages.

Summary updated

Abstract

In the last decade, the generalization and adaptation abilities of deep learning models were typically evaluated on fixed training and test distributions. Contrary to traditional deep learning, large language models (LLMs) are (i) even more overparameterized, (ii) trained on unlabeled text corpora curated from the Internet with minimal human intervention, and (iii) trained in an online fashion. These stark contrasts prevent researchers from transferring lessons learned on model generalization and adaptation in deep learning contexts to LLMs.To this end, our short paper introduces empirical observations that aim to shed light on further training of already pretrained language models. Specifically, we demonstrate that training a model on a text domain could degrade its perplexity on the test portion of the same domain. We observe with our subsequent analysis that the performance degradation is positively correlated with the similarity between the additional and the original pretraining dataset of the LLM. Our further token-level perplexity analysis reveals that the perplexity degradation is due to a handful of tokens that are not informative about the domain. We hope these findings will guide us in determining when to adapt a model vs when to rely on its foundational capabilities.
BibTeX
@inproceedings{oncel2024adaptation,
  author = {Firat Öncel and Matthias Bethge and Beyza Ermis and Mirco Ravanelli and Cem Subakan and Çagatay Yildiz},
  title = {{Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=GFEHh86PrJ},
}
Figure