TL;DR
This paper introduces the LP Data Pipeline, a novel, entirely CPU-based framework for efficiently creating high-quality, domain-specific datasets for Large Language Models (LLMs). It addresses the significant computational cost and time barriers associated with GPU-intensive data curation, making LLM development more accessible.
What Problem Does This Paper Address?
The creation of massive, high-quality datasets is crucial for training advanced Large Language Models (LLMs). Traditionally, this process has relied on resource-intensive, GPU-accelerated models for data extraction and filtering. While effective, this approach is time-consuming and costly, creating a significant barrier for organizations lacking substantial computational infrastructure. Furthermore, there's a growing demand for LLMs tailored to specific domains (e.g., finance, law) and languages, requiring specialized datasets that current general-purpose pipelines may not adequately provide. This paper aims to democratize LLM development by offering a lightweight, cost-effective, and purpose-driven data curation solution that runs entirely on CPUs.
Methods
The LP Data Pipeline is built upon four core principles:
- Fully CPU-Based Data Curation: The entire pipeline operates on CPUs, avoiding the need for expensive GPU infrastructure. It utilizes optimized and efficient CPU-based methods for data processing.
- Optimized Processing Order for Efficiency: Tasks are sequenced strategically. Initial filtering and basic cleansing are performed first, reserving more resource-intensive operations for later stages. This ensures that only higher-quality data undergoes advanced processing, saving time and resources.
- Continuous Knowledge Updating: The pipeline incorporates automated mechanisms for keeping datasets up-to-date by processing regular data dumps. Workflow management tools like Airflow automate job scheduling.
- Purpose-Driven Dataset Construction: The pipeline supports the creation of datasets tailored to specific domains and languages. It uses a FastText model for domain classification to extract data relevant to specialized applications.
Key stages within the pipeline include:
- Raw Text Extraction and URL Filtering: Extracts text from sources like CommonCrawl, removing boilerplate and irrelevant HTML.
- Language Identification: Uses a CPU-optimized FastText model to accurately identify and retain data in target languages.
- Line-Level Deduplication with Domain Grouping: Removes redundant lines by grouping documents by their domain URLs, which is more efficient than random grouping.
- Heuristic Filtering: Generates metadata (text length, stop word ratio, n-gram repetition, PII detection) and applies rule-based filtering based on predefined thresholds.
- Global Deduplication: Employs MinHash LSH for efficient document-level deduplication.
- Model-based Quality Filtering: Leverages FastText, trained on "Good" (Wikipedia, academic texts) and "Bad" (toxic content, informal posts) KenLM models, to assess data quality.
- Domain Classification: A FastText model classifies documents into specific industrial domains (finance, law, healthcare) or an "others" category.
Datasets
- CommonCrawl: Used as the primary source for raw web data. The paper specifically mentions using the CC-MAIN-2024-10 and CC-MAIN-2024-38 dumps.
- Wikipedia: Used as a source for high-quality data and for training the "Good KenLM" model.
- Academic Textbooks: Also used for training the "Good KenLM" model.
- Toxic Datasets: Used for training the "Bad KenLM" model.
The paper does not introduce new benchmarks but rather focuses on creating specialized datasets from existing sources.
Key Insights and Ideas
- CPU-centric LLM Data Curation is Feasible and Cost-Effective: The most significant insight is that high-quality LLM data can be curated effectively and affordably using only CPUs, challenging the prevailing notion that GPUs are indispensable for this task.
- Domain-Specific Datasets Significantly Enhance LLM Performance: The paper highlights the growing importance of tailoring LLMs to specific domains and languages, and the LP Data Pipeline directly addresses this need by enabling the creation of such specialized datasets.
- Strategic Processing Order is Key to Efficiency: By carefully ordering the data processing steps, the pipeline minimizes computational waste, ensuring that expensive operations are applied only to data that has already passed initial quality checks.
- "Good" vs. "Bad" KenLM Training for Quality Filtering: The innovative approach of training a quality filtering model using both high-quality and low-quality examples (KenLM models) provides a more robust and nuanced assessment of data suitability for LLM training.
Key Results
- Cost-Effectiveness: Processing a 4TB CommonCrawl dump (CC-MAIN-2024-10) on 128 machines with 8-core CPUs took 4 hours and 22 minutes, costing approximately $352.83. This is presented as a significant cost saving compared to GPU-intensive methods.
- Scalability: The pipeline demonstrated scalability by processing 10 CommonCrawl dumps, with results suggesting that the time and cost scale linearly.
- Domain-Specific Data Generation: The pipeline successfully generated domain-specific datasets for English and Korean, including Medical, Financial, and Legal corpora, with quantities detailed in Table 3. For instance, the English Medical Corpus contained 5.02 million documents and 4.27 billion tokens.
- Boilerplate Removal: Line-level deduplication with domain grouping effectively removed boilerplate content, as illustrated by sample removed lines in Table 4.
Limitations and Future Work
- Limited Language and Domain Support: The current language and domain classification is restricted to English, Korean, Japanese, and Thai, and specific domains like finance, law, and healthcare. Expanding this to more languages and domains, especially low-resource ones, is a key area for improvement.
- No Extensive LLM Training Experiments: Due to resource constraints, the authors did not conduct extensive LLM training experiments using the curated datasets. They plan to do this in future work to empirically validate the impact of the LP Data Pipeline on model performance.
- Future Work: The authors plan to conduct comprehensive evaluations to assess how the curated datasets influence LLM training outcomes and to expand the framework's language and domain coverage.
Summary updated