TL;DR
This paper introduces a novel method for selecting high-quality instruction data for training Large Language Models (LLMs) by injecting noise into the input embeddings and measuring the consistency of the model's output. This approach avoids relying on external models and effectively identifies data that aligns with the LLM's intrinsic knowledge, leading to significant performance improvements with less data.
What Problem Does This Paper Address?
Instruction tuning has become a crucial step in enhancing the capabilities of Large Language Models (LLMs), enabling them to perform a wide range of natural language processing tasks. While early research focused on the quantity of instruction data, recent work highlights the importance of data quality. However, existing methods for selecting high-quality data often rely on external models or predefined rules, which may not fully capture the intrinsic alignment between the pre-trained LLM and the instruction data. This misalignment can limit the model's performance, as it might not effectively leverage its existing knowledge. The core challenge this paper addresses is: How to select instruction data that genuinely aligns with an LLM's own preferences and intrinsic knowledge to achieve optimized instruction tuning?
Methods
The paper proposes a novel strategy called Consistency Verification of Noise Injection. The key idea is to leverage the LLM's own internal consistency as a measure of data quality.
Here's how it works:
- Noise Injection: Instead of modifying the entire instruction, noise is injected into the embedding space of the input part of the instruction. This is done using Gaussian noise, with parameters (mean and variance) adaptively calculated based on the input embeddings to ensure a meaningful semantic shift.
- Consistency Calculation: The pre-trained LLM processes both the original instruction and the noisy version. The probability distribution of the model's output is then compared between these two versions. A low KL divergence between the output distributions indicates high consistency, suggesting that the instruction is of high quality and aligns well with the model's learned knowledge.
- Diversity Selection: To prevent the selected dataset from becoming redundant or biased towards a narrow set of topics, the method incorporates two diversity strategies:
- Inter-class diversity: Uses K-means clustering on semantic embeddings to ensure a broad coverage of different categories.
- Intra-class diversity: Assesses similarity within classes and selects less similar data points to reduce redundancy.
Datasets
The paper utilizes several datasets for training and evaluation:
- Alpaca: A 52K instruction dataset created by Stanford University, used as the primary dataset for filtering and experiments.
- Dolly: A manually crafted instruction dataset used to test the method's generalization across different stylistic characteristics.
- FLAN: A dataset of instruction-formatted tasks, also used for testing generalization.
- MMLU (Massive Multitask Language Understanding): Used to measure the model's ability to handle interdisciplinary knowledge.
- Humaneval: Used to evaluate the model's proficiency in code comprehension and generation.
- GSM-8K: Used to assess the model's aptitude in resolving mathematical problems.
- CommonsenseQA: Used to examine the model's mastery of common sense knowledge.
- NaturalQuestions: Used to evaluate performance in understanding and answering questions involving world knowledge.
- IFEVAL: Used to measure instruction adherence across diverse task formats.
- Vicuna testset: Used for evaluating the model's ability to express internal knowledge.
Key Insights and Ideas
- Noise as a Quality Signal: The core insight is that injecting noise into instruction embeddings can act as a powerful signal for data quality. High-quality instructions are those that remain consistent in their output probability distribution even when their input is slightly perturbed.
- Intrinsic Alignment is Key: The method's success stems from its ability to identify data that aligns with the LLM's own pre-trained knowledge and preferences, rather than relying on external judgments. This is a significant departure from many existing methods.
- Efficiency Gains: The proposed method can achieve superior performance by selecting only a small fraction (5%-15%) of the original dataset, drastically reducing training costs and time.
- Diversity is Crucial for Generalization: Simply selecting high-consistency data can lead to a lack of diversity. The combination of consistency and diversity selection is essential for robust performance across various tasks and domains.
- Noise Intensity Matters: There's an optimal range for noise intensity. Too little noise doesn't provide enough signal, while too much can overwhelm the model, making it difficult to discern relevant problems.
Key Results
- The proposed method significantly outperforms established baselines and full-data training across various LLM scales and architectures.
- With only 5%-15% of the data, the method achieves performance comparable to or better than training on the entire dataset.
- For example, on the Llama2-7B model, the method achieved a +2.93 average score improvement over baselines while using a small subset of data.
- The method demonstrates strong generalization capabilities, performing well on diverse datasets like Dolly and FLAN, and on newer, larger models like Llama3-8B and Qwen2-7B/2.5-14B.
- The selected data tends to favor creative and interpretive tasks (e.g., "generate," "write") over revision tasks (e.g., "rewrite," "edit"), suggesting a focus on instructions with higher semantic information content.
Limitations and Future Work
- Inference Cost: The method introduces additional inference costs due to the need to process both original and noisy versions of the data.
- Model Type Similarity: The authors observe that models of the same type tend to select similar filtered data, likely due to similar pre-training data. They suggest that future work could explore filtering on smaller models and applying these results to larger, similar models.
- Further Exploration of Noise: While the paper explores noise intensity, further research could investigate other noise types or injection strategies.
- Adaptive Filtering: The authors suggest that future research could focus on more sophisticated adaptive filtering strategies that consider both quality and diversity more dynamically.
Summary updated