Introduction
The standard "Pretrain then Fine-tune" pipeline has become the cornerstone of modern NLP. Models first learn general language features from massive, noisy datasets before specializing on smaller, high-quality "in-domain" datasets. However, generic pretraining data is often too broad. Data selection—filtering pretraining sets for domain-relevant examples—serves as a powerful intermediate phase to bridge this gap. This paper investigates the "complementarity" between data selection and final fine-tuning, revealing that aggressive selection can sometimes be counterproductive.
The Three-Phase Pipeline
The authors evaluate a training pipeline consisting of three distinct phases:
- Pretraining (PT): Training on a full out-of-domain (OOD) dataset.
- Selected Data Training: Using an intermediate classifier to rank OOD data and training further on the top-ranked subset.
- Fine-tuning (FT): Final training on the target in-domain dataset.
Two primary selection strategies are compared: Contrastive Data Selection (CDS), which uses likelihood differences between in-domain and OOD language models, and Discriminative Classifiers (DC), which range from Logistic Regression to fine-tuned BERT models trained to distinguish domain membership.
The Complementarity Trap
A key finding is the "complementarity trap": more aggressive data selection does not always yield better final performance. If a selection classifier overfits the small in-domain set, it selects data that is redundant with the fine-tuning set. This redundancy erodes the unique benefits of the final fine-tuning stage. Effective selection must find "near-domain" examples that provide novel information rather than just mimicking the fine-tuning data.
Experimental Results
The study covers Language Modeling (LM) on the Yelp corpus and Machine Translation (MT) for English-to-German (En-De) and English-to-Russian (En-Ru).
Machine Translation Performance
As shown in the results table, the BERT-based discriminative classifier consistently outperforms other methods. For En-De translation, the full PT + DC-BERT + FT pipeline achieved a BLEU score of 27.78, a significant improvement over the 26.89 BLEU achieved by standard fine-tuning alone. The BERT-based classifier outperformed the traditional CDS method with over 99% statistical significance across multiple metrics.
Efficiency and Scaling
Data selection also offers a significant efficiency shortcut. Applying selection to an early pretraining checkpoint can reach target loss levels up to 3.5x faster than continuing with generic pretraining. Furthermore, there is a trade-off between selection size and speed: smaller sets (e.g., 50k sentences) offer rapid initial gains but lower performance ceilings compared to larger sets (e.g., 1 million sentences).
Conclusion
Discriminative classifiers, particularly those based on BERT, are superior to traditional contrastive methods for data selection. When properly balanced to avoid the complementarity trap, data selection and fine-tuning work together to significantly improve domain adaptation performance and training efficiency.
Summary updated