Abstract
Modern Large Language Model (LLM) development involves a complex multi-stage pipeline: pre-training, continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL). Despite the success of these models, the interplay between these stages remains a "black box." EvoLM presents a systematic study of over 100 language models (1B and 4B parameters) trained from scratch to map these dynamics. We uncover critical trade-offs, such as the "over-training trap" where excessive pre-training degrades downstream reasoning, and demonstrate that RL primarily acts as a confidence booster for existing knowledge.
Introduction
The development of LLMs is often treated as a relay race, yet the impact of early-stage decisions on final performance is poorly understood. Most research relies on pre-trained base models, preventing control over variables like data composition or learning rate schedules. EvoLM addresses this by providing a transparent, end-to-end investigation into the training process, from initial tokens to final RL alignment.
Methodology
The researchers developed a reproducible pipeline based on the Llama-2 architecture, training a suite of models from 0.5B to 4B parameters. The training process consists of four distinct stages:
- Pre-training: Models were trained on up to 320B tokens using FineWeb-Edu.
- Continued Pre-training (CPT): Focused on mathematical reasoning using FineMath.
- Supervised Fine-Tuning (SFT): Applied to math-heavy datasets (e.g., GSM8K, MATH).
- Reinforcement Learning (RL): Utilized PPO with verifiable rewards to refine problem-solving.
To ensure traceability, each model is identified by a "compact model signature" detailing the data and compute used at every stage.
Key Insights and Training Dynamics
The Over-training Trap
A significant finding is that while increasing pre-training tokens improves performance on "cloze" tasks, it can lead to diminishing or negative returns for downstream reasoning. Performance typically saturates when training data reaches 80x to 160x the model's parameter count. Furthermore, under fixed compute budgets, a smaller model trained to completion often outperforms a larger, under-trained counterpart.
Mitigating Catastrophic Forgetting in CPT
Transitioning to specialized domains like mathematics often causes models to lose general knowledge. The study found that "replaying" just 5% of the original pre-training data during the CPT stage effectively mitigates this catastrophic forgetting.
The Role of Reinforcement Learning
RL is often viewed as a way to teach models new skills. However, EvoLM suggests RL acts more like a "magnifying glass" or "confidence booster." It sharpens the model's preference for correct solutions it already had the capacity to generate, rather than creating entirely new reasoning capabilities. This benefit is most pronounced when the model has not been "over-memorized" during the SFT stage.
Evaluation Metrics: Perplexity vs. ORM
The study highlights a disconnect between standard metrics and actual performance. For post-trained models, validation perplexity correlates poorly with problem-solving accuracy. In contrast, Outcome Reward Model (ORM) scores serve as much more reliable predictors of downstream success.
Results and Conclusion
EvoLM’s 1B and 4B models achieved performance competitive with much larger open-source models (e.g., TinyLlama-1B, Qwen1.5-4B) despite being trained on significantly fewer tokens (320B vs. 2T-3T). However, the study also notes that increasing SFT epochs, while beneficial for in-domain tasks, can cause a performance crash in out-of-domain areas like coding or logic due to over-specialization.
By open-sourcing the 100+ model suite and the associated data/pipelines, EvoLM provides a foundation for the community to further explore the intricate dynamics of language model training.
Summary updated