CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models

Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, Fei Tan

Abstract

Large Language Models (LLMs) often require adaptation to specialized domains through Continual Pre-training (CPT). However, the process of mixing new domain-specific data with general corpus data has historically relied on heuristic methods, often leading to inefficient training or the degradation of general capabilities (catastrophic forgetting). This paper introduces the Critical Mixture Ratio (CMR), a predictable threshold that defines the optimal balance between domain acquisition and general knowledge retention. By establishing a power-law relationship between loss, mixture ratio, and training tokens, we provide a mathematical framework to optimize CPT across various model scales and data volumes.

Introduction

To enhance the utility of LLMs in specialized fields like law, medicine, or finance, researchers employ Continual Pre-training (CPT). This involves training an existing model on a new domain-specific corpus. To prevent the model from losing its original reasoning and linguistic capabilities, a portion of the original general training data is typically "replayed" alongside the new data.

Choosing the right mixture ratio ($R$) is a critical challenge. An excessive amount of domain data triggers catastrophic forgetting, while too little results in slow learning and wasted compute. We formalize this trade-off by identifying the CMR: the maximum proportion of domain data a model can ingest while keeping the increase in general loss within a predefined tolerance ($\epsilon$).

Methodology: The CMR Scaling Law

We treat the data mixture problem as a constrained optimization task. Using Lagrange multipliers, we aim to minimize loss on the new domain subject to the constraint that the general knowledge loss does not exceed a specific threshold.

Power-Law Modeling

Our research reveals that the relationship between loss, the mixture ratio ($R$), and the number of training tokens ($T$) follows a power-law distribution, consistent with established scaling laws in the field. To account for the "stability gap"—a phenomenon where general loss spikes at the onset of CPT before stabilizing—we utilize a two-term polynomial fitting function. This approach allows for high-precision modeling of training trajectories, enabling researchers to predict how a model will behave long before the training run is complete.

Experimental Setup

We conducted extensive experiments using a general corpus of 220 billion tokens (comprising Chinese, English, and code) and two specialized domains:

  1. Finance: Financial news, policy documents, and research reports.
  2. Academic Papers: A subset of Arxiv publications.

We trained a series of Llama-style models with parameter counts ranging from 460M to 3.1B to observe how the CMR scales with model size.

Key Insights

1. Model Capacity and Resilience

A primary finding is that larger models exhibit higher resilience to new data. For the Finance domain, the CMR increased significantly with model size:

  • 460M Model: 29.8% CMR
  • 3.1B Model: 47.8% CMR

This suggests that larger models possess a greater "buffer" or capacity to integrate new knowledge without disrupting existing neural representations.

2. Domain Similarity

The CMR is not a universal constant but is influenced by the similarity between the general corpus and the target domain. Because our general data already included academic content, the models could handle a higher mixture ratio of Academic Papers (36.7%) compared to Finance (29.8%) at the same scale.

3. Predictability and Efficiency

The proposed scaling law demonstrates remarkable accuracy. In our tests, the fitting accuracy ($R^2$) frequently exceeded 0.99, with relative errors as low as 0.02%. This predictability allows for "small-to-large" extrapolation: engineers can run cheap, small-scale experiments to determine the optimal CMR for massive, expensive production models.

Discussion and Limitations

While the CMR scaling law provides a robust framework for CPT, several areas remain for future exploration:

  • Scaling to Frontier Models: Our experiments reached 3.1B parameters. Further validation is required to confirm these laws hold for models in the 70B+ range.
  • Multi-Domain Complexity: Real-world CPT often involves blending multiple specialized domains simultaneously, which may require more complex multi-objective optimization.
  • Loss vs. Downstream Metrics: Our study uses cross-entropy loss as the primary metric. While loss is a reliable proxy for performance, future work should correlate CMR directly with downstream benchmarks, such as professional certification exams.

Conclusion

The discovery of the CMR scaling law transforms the "art" of data mixing into a rigorous science. By providing a mathematical recipe for continual pre-training, this framework ensures that LLMs can be specialized efficiently without sacrificing the general intelligence that makes them powerful. This offers a practical guideline for organizations looking to build domain-specific expertise into their AI systems while managing finite computational resources.

Summary updated

Abstract

Large Language Models (LLMs) excel in diverse tasks but often underperform in specialized fields due to limited domain-specific or proprietary corpus. Continual pre-training (CPT) enhances LLM capabilities by imbuing new domain-specific or proprietary knowledge while replaying general corpus to prevent catastrophic forgetting. The data mixture ratio of general corpus and domain-specific corpus, however, has been chosen heuristically, leading to sub-optimal training efficiency in practice. In this context, we attempt to re-visit the scaling behavior of LLMs under the hood of CPT, and discover a power-law relationship between loss, mixture ratio, and training tokens scale. We formalize the trade-off between general and domain-specific capabilities, leading to a well-defined Critical Mixture Ratio (CMR) of general and domain data. By striking the balance, CMR maintains the model's general ability and achieves the desired domain transfer, ensuring the highest utilization of available resources. Considering the balance between efficiency and effectiveness, CMR can be regarded as the optimal mixture ratio. Through extensive experiments, we ascertain the predictability of CMR, propose CMR scaling law and have substantiated its generalization. These findings offer practical guidelines for optimizing LLM training in specialized domains, ensuring both general and domain-specific performance while efficiently managing training resources.
BibTeX
@inproceedings{gu2024cmr,
  author = {Jiawei Gu and Zacc Yang and Chuanghao Ding and Rui Zhao and Fei Tan},
  title = {{CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=mahelL6MBn},
}
Figure