Abstract
Domain-Adaptive Pretraining (DAPT) is a critical step in specializing large language models for specific tasks, yet it often involves processing massive, uncurated datasets that demand significant computational resources. This paper introduces TextGram, an adaptive data selection strategy that combines n-gram frequency analysis with graph-based ranking to identify the most relevant training samples from general-purpose corpora. By focusing on high-quality, domain-aligned data, TextGram improves model performance on downstream tasks while reducing the environmental impact of training.
Introduction
The rapid advancement of Large Language Models (LLMs) has come at a significant environmental cost. Training a single large transformer model can emit as much $CO_2$ as five cars over their entire lifetimes. When adapting these models to specific domains—such as medical records or movie reviews—researchers typically employ Domain-Adaptive Pretraining (DAPT). However, standard DAPT often relies on massive, general-purpose datasets containing substantial "noise" irrelevant to the target task. This inefficiency leads to wasted compute and potential "negative transfer," where irrelevant data degrades model performance. TextGram addresses this by intelligently selecting a subset of data that is most relevant to the target domain.
Methodology
The TextGram framework is an adaptive data selection pipeline designed to bridge the gap between general-purpose corpora and domain-specific tasks. The process involves four primary stages:
- N-Gram Filtering: The algorithm identifies frequent n-grams in the target in-domain dataset and extracts sentences from the general out-domain corpus that contain these sequences.
- Paraphrase Mining: Similarity scores are calculated between the selected out-domain sentences and the in-domain data to determine contextual relevance.
- Graph-Based Ranking (TextRank): A graph is constructed where sentences serve as nodes and similarity scores represent edges. The PageRank algorithm is applied to rank sentences based on their centrality and relevance to the domain.
- Selection: The top $N$ ranked sentences are curated into a "Selected Corpus" for pretraining.
Unlike traditional unsupervised TextRank, TextGram is "domain-aware," using the target dataset to guide the graph construction and ranking process.
Experimental Setup
To evaluate TextGram, the authors utilized two primary datasets representing general and domain-specific text.
The RealNews dataset, comprising 1 million news articles, served as the out-domain source. The IMDb Movie Reviews dataset, containing 50,000 reviews, served as the in-domain target for sentiment analysis. The goal was to select the most relevant news articles to pretrain a model for better performance on IMDb sentiment classification.
Results and Discussion
TextGram was compared against several baseline selection methods, including random selection, perplexity-based selection (Cross-Entropy), and standard TextRank.
Experimental results show that TextGram achieved an accuracy of 91.02%, outperforming the uncurated baseline (90.40%), Random Selection (90.77%), and standard TextRank (90.60%). While the absolute accuracy gain is approximately 1%, the primary achievement is reaching this performance level while using only a fraction of the original 1M news articles. This efficiency translates directly into reduced "GPU-days" and a smaller carbon footprint, challenging the notion that more data is always superior.
Conclusion
TextGram represents a step toward "Green AI" by making domain adaptation more efficient. By leveraging n-gram seeding and graph-based ranking, it ensures that pretraining data is contextually aligned with the target task. Future work will explore more complex graph implementations and test the strategy across a broader range of downstream tasks beyond binary classification.
Summary updated