TL;DR
This paper introduces MANTA, an automated pipeline that generates high-quality, large-scale instruction fine-tuning datasets from massive web corpora. By extracting structured syllabi and leveraging LLMs, MANTA significantly outperforms existing methods, particularly in knowledge-intensive tasks, offering a scalable and efficient way to create diverse instruction datasets.
What Problem Does This Paper Address?
The creation of high-quality, large-scale instruction fine-tuning datasets for Large Language Models (LLMs) is crucial for improving their capabilities. However, directly constructing these datasets, which consist of user queries and assistant responses, is a labor-intensive and expensive process requiring significant human intervention. While automated pipelines have emerged, they often face limitations in terms of scalability, diversity, or reliance on carefully curated seed examples. This paper addresses the challenge of efficiently generating diverse, scalable, and high-quality instruction datasets from vast web corpora with minimal human effort.
Methods
MANTA employs a two-stage pipeline:
- Syllabus-based Generation Stage:
- Web Document to Structured Syllabi: Massive web corpora are processed. LLMs are used to encapsulate the core knowledge of each document into a JSON-formatted "Syllabus." This syllabus includes attributes like Domain, Subject, Unit, Lesson, Class Session, and Key Concept. This stage is fully automated and aims to reflect the diverse topic distribution of the web.
- Domain Attribute Enhancement: A bottom-up categorization approach using k-means clustering on text embeddings is applied to assign a "Domain" attribute to syllabi, further organizing the knowledge.
- Advanced Generation Stage:
- Syllabus-based Instruction Generation: LLMs are prompted to generate query-response pairs from the structured syllabi. This involves first generating a query and then using that query to generate a response.
- Difficulty Control (Syllabus Fusion): To increase instruction difficulty, multiple syllabi from the same domain are fused. An LLM generates a fused syllabus, which is then used to create more complex instructions.
- Multi-turn Expansion: To simulate real-world conversations, the dataset is expanded to include multi-turn scenarios. Four types of follow-up queries are defined: Deepening Question, Task Extension, Style Transfer, and Modification. LLMs generate these follow-up queries based on the conversation context and a given condition.
Datasets
- Training Data: The MANTA pipeline is trained on 1 million data points generated by MANTA itself (MANTA-1M). For evaluation, datasets like MAmmoTH2-1M, MAGPIE-PRO-1M, and ORCA 3-1M are used for comparison.
- Web Corpora for Syllabus Extraction: Wikipedia, Fine Web, and Knowledge Pile are used to extract syllabi, totaling approximately 20 million documents.
- Evaluation Benchmarks:
- Academic General: MMLU, MMLU-PRO, ARC-C.
- Domain Specific: GSM8K, MATH 500, THEOREMQA, GPQA, HUMANEVAL.
- Instruction Following: MT-BENCH, ALPACAEVAL 2 LC.
Key Insights and Ideas
- Syllabi as Knowledge Encapsulators: The core idea of distilling web content into structured syllabi is highly effective for generating coherent and knowledge-rich instructions. This approach allows for a more systematic capture of information compared to direct extraction from raw text.
- Automated Diversity and Scalability: MANTA's pipeline is designed to be fully automated, enabling it to scale to massive web corpora and naturally reflect the diverse topic distribution present online, overcoming limitations of manually curated datasets.
- Syllabus Fusion for Difficulty: Combining syllabi from the same domain is a simple yet effective method to generate more challenging instructions without explicit human effort in designing difficulty.
- Query Composition is Key: The experiments highlight that the way queries are composed is a dominant factor in model performance, even more so than the sophistication of the response generator. MANTA's syllabus-driven query generation excels in this regard.
- Generalization Across Domains: Models fine-tuned on MANTA-1M demonstrate strong performance not only in academic knowledge but also across various domains like mathematics and coding, suggesting the dataset's broad applicability.
Key Results
- Fine-tuning on the MANTA-1M dataset significantly outperforms other massive dataset generation methodologies across various benchmarks.
- Notable performance gains are observed in knowledge-intensive tasks like MMLU and MMLU-PRO.
- MANTA-1M also shows superior performance in mathematics (MATH-500) and coding tasks.
- The difficulty analysis shows MANTA has the highest average difficulty level (8.52) among compared datasets, achieved by simply combining syllabi.
- The pipeline is efficient, requiring only 3 LLM calls per instruction (syllabus extraction, instruction generation, response generation), compared to more complex multi-stage processes in other methods.
Limitations and Future Work
- Corpus Size: The experiments were conducted with a 20 million document corpus. Future work could explore how MANTA performs with larger corpora and different raw data formats.
- Model Specificity: Experiments were conducted with specific LLMs. Future research could explore the pipeline's effectiveness across a wider range of models.
- Harmful Content: While the dataset is reported to be 95% safe, a small percentage (3%) of "Specialized Advice" responses were flagged as potentially harmful, particularly concerning medical, legal, or financial advice. The authors acknowledge the need for careful labeling and ethical considerations.
- Domain-Specific Syllabus Impact: While experiments show that using domain-specific syllabi can enhance performance on related downstream tasks, further research could explore this more deeply.
Summary updated