Rule-based Data Selection for Large Language Models

Xiaomin Li, Mingye Gao, Zhiwei Zhang, Chang Yue, Hong Hu

TL;DR

This paper introduces a novel, automated framework for selecting high-quality data to train Large Language Models (LLMs). By using LLMs to generate diverse quality rules and then employing a mathematical technique called Determinantal Point Process (DPP) to select the most independent rules, the authors achieve better data selection, leading to improved LLM performance across various domains.

What Problem Does This Paper Address?

Training LLMs requires vast amounts of high-quality data. While recent work has explored using LLMs themselves to rate data quality based on human-defined rules, these methods often rely on subjective human heuristics, lack principled ways to evaluate the rules themselves, and struggle to adapt to new tasks. A key challenge is that many rules can be redundant or correlated, leading to biased data selection. The authors aim to address this by developing a principled, automated system that can generate, evaluate, and select a diverse and independent set of rules for data selection, ultimately leading to better LLM training.

Methods

The core of the paper's contribution lies in its automated, multi-stage pipeline:

  1. Rule Generation: LLMs (specifically GPT-4) are prompted to generate a large set of diverse rules for data quality assessment, covering various aspects. These rules are then manually reviewed for clarity.
  2. Rule-based Rating: Another LLM (Llama3-8B-Instruct) is used to rate a batch of data samples according to the generated rules, creating a score matrix.
  3. Rule Selection using DPP: This is the novel part. Instead of randomly picking rules or using fixed sets, the authors use the Determinantal Point Process (DPP). DPP is a probabilistic model that favors diverse subsets. Here, it's applied to the score vectors of the rules. The goal is to select a subset of rules whose score vectors are as orthogonal (uncorrelated) as possible. This is quantified by a metric based on the deviation of the correlation matrix from the identity matrix. DPP helps select a compact set of independent rules that capture distinct aspects of data quality.
  4. Data Selection: The selected, orthogonal rules are then used to score the entire dataset. Instead of a simple top-k selection, a stochastic sampling strategy is employed to balance quality and diversity.
  5. Application: The selected data is used for downstream tasks, primarily LLM fine-tuning.

Datasets

The authors evaluated their framework on several datasets:

  • IMDB: A dataset for sentiment analysis.
  • MedQA (Medical): A dataset for medical question answering.
  • GSM8K (Math): A dataset for elementary math problems.
  • MBPP (Code): A dataset for code generation.
  • CommonCrawl: A large, general-purpose web corpus used for LLM pre-training.

For evaluation, they also used human-annotated ground truth ratings on subsets of these datasets.

Key Insights and Ideas

  • Orthogonality as a Metric for Rule Quality: The paper's central idea is that rule diversity and independence, measured by the orthogonality of their score vectors, are crucial for effective data selection. Correlated rules can lead to redundant signals and biased data.
  • DPP for Rule Selection: Applying DPP to rule score vectors is a principled way to select a diverse and non-redundant subset of rules, moving beyond heuristic or random selection.
  • Automated Pipeline: The framework is fully automated, from rule generation to data selection, significantly reducing manual effort.
  • Task-Specific Rules Matter: The effectiveness of rules is shown to be context-dependent. The authors' method adapts rule selection to the specific downstream task, unlike static, pre-defined rule sets.
  • Quality Over Quantity: The results reinforce the idea that carefully selected high-quality data is more important than simply having a large volume of data for LLM training.

Key Results

  • Improved Rating Accuracy: The DPP-based rule selection consistently achieved lower Mean Squared Error (MSE) compared to baselines like rule-free rating, human-created rules (QuRating), and randomly selected rules, when compared against ground truth ratings.
  • Better LLM Performance: LLMs fine-tuned on data selected by the proposed framework showed significant improvements in performance across IMDB, Medical, Math, and Code domains, outperforming baselines like uniform sampling, DSIR, LESS, and QuRating. For example, in Table 1, DPP 10 Rules consistently achieved higher scores across various subtasks compared to other methods.
  • Rule Diversity Correlates with Accuracy: Experiments showed a strong positive correlation between rule diversity (lower rule correlation) and rating accuracy (lower MSE).
  • DPP Outperforms GPT-Generated "Uncorrelated" Rules: Even when GPT was prompted to generate "uncorrelated" rules, the DPP-selected rules (based on task-specific scores) still exhibited lower correlation and led to better performance, highlighting the importance of task-aware rule selection.

Limitations and Future Work

  • Hyperparameter Tuning: While the framework is robust, the optimal number of selected rules (r) is a hyperparameter. The authors suggest further exploration of its optimal values.
  • Data Sampling Variations: The stochastic sampling method could be further explored with variations like Gumbel top-k trick and other sampling strategies.
  • Rule Format: The current framework focuses on natural language rules. Integrating rules in other formats is a potential future direction.
  • Rule Evaluation Metrics: While orthogonality is used, exploring other metrics for rule evaluation could be beneficial.
  • Human vs. LLM Rule Generation: While LLMs are shown to be effective rule generators, further research could explore hybrid approaches or more sophisticated human-in-the-loop mechanisms for rule refinement.
  • Generalization to Other Tasks: Extending the framework to tasks like dialog safety or creative writing would require new evaluation methodologies.

Summary updated

Abstract

The quality of training data significantly impacts the performance of large language models (LLMs). There are increasing studies using LLMs to rate and select data based on several human-crafted metrics (rules). However, these conventional rule-based approaches often depend too heavily on human heuristics, lack effective metrics for assessing rules, and exhibit limited adaptability to new tasks. In our study, we introduce an innovative rule-based framework that utilizes the orthogonality of score vectors associated with rules as a novel metric for rule evaluations. Our approach includes an automated pipeline that first uses LLMs to generate a diverse set of rules, encompassing various rating dimensions to evaluate data quality. Then it rates a batch of data based on these rules and uses the determinantal point process (DPP) from random matrix theory to select the most orthogonal score vectors, thereby identifying a set of independent rules. These rules are subsequently used to evaluate all data, selecting samples with the highest average scores for downstream tasks such as LLM training. We verify the effectiveness of our method through two experimental setups: 1) comparisons with ground truth ratings and 2) benchmarking LLMs trained with the chosen data. Our comprehensive experiments cover a range of scenarios, including general pre-training and domain-specific fine-tuning in areas such as IMDB, Medical, Math, and Code. The outcomes demonstrate that our DPP-based rule rating method consistently outperforms other approaches, including rule-free rating, uniform sampling, importance resampling, and QuRating, in terms of both rating precision and model performance.
BibTeX
@misc{li2024rule,
  author = {Xiaomin Li and Mingye Gao and Zhiwei Zhang and Chang Yue and Hong Hu},
  title = {{Rule-based Data Selection for Large Language Models}},
  year = {2024},
  eprint = {2410.04715},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.04715},
}
Figure