EvoLM: In Search of Lost Language Model Training Dynamics

Zhenting Qi, Fan Nie, Alexandre Alahi, James Zou, Himabindu Lakkaraju, Yilun Du, Eric P. Xing, Sham M. Kakade, Hanlin ZhangarXiv 2025 · DOI · arXiv

Abstract

Modern Large Language Model (LLM) development involves a complex multi-stage pipeline: pre-training, continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL). Despite the success of these models, the interplay between these stages remains a "black box." EvoLM presents a systematic study of over 100 language models (1B and 4B parameters) trained from scratch to map these dynamics. We uncover critical trade-offs, such as the "over-training trap" where excessive pre-training degrades downstream reasoning, and demonstrate that RL primarily acts as a confidence booster for existing knowledge.

Introduction

The development of LLMs is often treated as a relay race, yet the impact of early-stage decisions on final performance is poorly understood. Most research relies on pre-trained base models, preventing control over variables like data composition or learning rate schedules. EvoLM addresses this by providing a transparent, end-to-end investigation into the training process, from initial tokens to final RL alignment.

Methodology

The researchers developed a reproducible pipeline based on the Llama-2 architecture, training a suite of models from 0.5B to 4B parameters. The training process consists of four distinct stages:

  1. Pre-training: Models were trained on up to 320B tokens using FineWeb-Edu.
  2. Continued Pre-training (CPT): Focused on mathematical reasoning using FineMath.
  3. Supervised Fine-Tuning (SFT): Applied to math-heavy datasets (e.g., GSM8K, MATH).
  4. Reinforcement Learning (RL): Utilized PPO with verifiable rewards to refine problem-solving.
The EvoLM multi-stage training and evaluation pipeline.

To ensure traceability, each model is identified by a "compact model signature" detailing the data and compute used at every stage.

Key Insights and Training Dynamics

The Over-training Trap

A significant finding is that while increasing pre-training tokens improves performance on "cloze" tasks, it can lead to diminishing or negative returns for downstream reasoning. Performance typically saturates when training data reaches 80x to 160x the model's parameter count. Furthermore, under fixed compute budgets, a smaller model trained to completion often outperforms a larger, under-trained counterpart.

Mitigating Catastrophic Forgetting in CPT

Transitioning to specialized domains like mathematics often causes models to lose general knowledge. The study found that "replaying" just 5% of the original pre-training data during the CPT stage effectively mitigates this catastrophic forgetting.

Impact of pre-training tokens on In-Domain (ID) and Out-of-Domain (OOD) performance across SFT and RL stages.

The Role of Reinforcement Learning

RL is often viewed as a way to teach models new skills. However, EvoLM suggests RL acts more like a "magnifying glass" or "confidence booster." It sharpens the model's preference for correct solutions it already had the capacity to generate, rather than creating entirely new reasoning capabilities. This benefit is most pronounced when the model has not been "over-memorized" during the SFT stage.

Evaluation Metrics: Perplexity vs. ORM

The study highlights a disconnect between standard metrics and actual performance. For post-trained models, validation perplexity correlates poorly with problem-solving accuracy. In contrast, Outcome Reward Model (ORM) scores serve as much more reliable predictors of downstream success.

Results and Conclusion

EvoLM’s 1B and 4B models achieved performance competitive with much larger open-source models (e.g., TinyLlama-1B, Qwen1.5-4B) despite being trained on significantly fewer tokens (320B vs. 2T-3T). However, the study also notes that increasing SFT epochs, while beneficial for in-domain tasks, can cause a performance crash in out-of-domain areas like coding or logic due to over-specialization.

By open-sourcing the 100+ model suite and the associated data/pipelines, EvoLM provides a foundation for the community to further explore the intricate dynamics of language model training.

Summary updated

Abstract

Modern language model (LM) training has been divided into multiple stages, making it difficult for downstream developers to evaluate the impact of design choices made at each stage. We present EvoLM, a model suite that enables systematic and transparent analysis of LMs' training dynamics across pre-training, continued pre-training, supervised fine-tuning, and reinforcement learning. We train over 100 LMs with 1B and 4B parameters from scratch, and evaluate both upstream (language modeling) and downstream (problem-solving) capabilities, including considerations of both in-domain and out-of-domain generalization. Key insights highlight the diminishing returns from excessive pre-training and post-training, the importance and practices of mitigating forgetting during domain-specific continued pre-training, the crucial role of continued pre-training in bridging pre-training and post-training phases, and various intricate trade-offs when configuring supervised fine-tuning and reinforcement learning. To facilitate open research and reproducibility, we release all pre-trained and post-trained models, training datasets for all stages, and our entire training and evaluation pipeline.
BibTeX
@misc{qi2025evolm,
  author = {Zhenting Qi and Fan Nie and Alexandre Alahi and James Zou and Himabindu Lakkaraju and Yilun Du and Eric P. Xing and Sham M. Kakade and Hanlin Zhang},
  title = {{EvoLM: In Search of Lost Language Model Training Dynamics}},
  year = {2025},
  eprint = {2506.16029},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.16029},
}
Figure