CritiQ: Mining Data Quality Criteria from Human Preferences

Honglin Guo, Kai Lv, Qipeng Guo, Tianyi Liang, Zhiheng Xi, Demin Song, Qiuyinzhe Zhang, Yu Sun, Kai Chen, Xipeng Qiu, Tao Gui

TL;DR

This paper introduces CritiQ, a novel method for automatically mining data quality criteria from human preferences, significantly reducing the need for extensive manual annotation. CritiQ uses an agent-based workflow to derive interpretable criteria, which are then used to train a scorer for efficient data selection, leading to improved LLM performance on downstream tasks.

What Problem Does This Paper Address?

Training large language models (LLMs) effectively hinges on the quality of the data they are trained on. However, existing methods for selecting high-quality data are often labor-intensive and prone to biases. These methods include manually designing heuristics, relying on model perplexity, training classifiers, or using complex prompt engineering. All these approaches demand significant expert knowledge and human annotation effort, making them costly, time-consuming, and potentially introducing inherent biases that can hinder LLM performance. Furthermore, quality standards can vary significantly across different domains, making a one-size-fits-all approach difficult.

Methods

CritIQ tackles this problem with a two-pronged approach: CritIQ Flow for mining quality criteria and CritIQ Scorer for efficient data selection.

CritIQ Flow: This is an agent-based workflow that automatically extracts quality criteria from human preferences. It starts with a small set of human-annotated data pairs (~30). A "manager" agent iteratively refines and generates new criteria, while "worker" agents make pairwise judgments on data based on these criteria. A key innovation is the use of a "knowledge base" that compiles quality criteria from previous research, providing a strong starting point. The process also incorporates a "reflection" mechanism where agents analyze incorrect judgments to improve criteria. CritIQ Flow aims to produce verbal, interpretable criteria that are more reusable than heuristic-based methods.

CritIQ Scorer: Once CritIQ Flow has derived the quality criteria, these are used to annotate a much larger set of data pairs. This annotated data is then used to train a lightweight scoring model, CritIQ Scorer. This scorer can then efficiently assign quality scores to the entire dataset, enabling the selection of a high-quality subset.

Datasets

The paper focuses on three domains: code, math, and logic.

  • Code: Stack v2 (non-code subset)
  • Math: OpenWebMath (non-code subset)
  • Logic: Zyda-2

The paper uses these datasets to create three splits:

  • Dhuman: A small set of human-annotated pairs (~30) for training CritIQ Flow.
  • Dagent: A larger set of pairs annotated by worker agents for training CritIQ Scorer.
  • Dtest: Human-annotated test sets for evaluating the accuracy of the mined criteria.

Key Insights and Ideas

  • Human preferences are key: The core idea is that directly learning from human judgments, even a small amount, is more effective and less biased than relying on predefined heuristics or model-based metrics.
  • Interpretable criteria matter: CritIQ generates verbal criteria that are more understandable and reusable across different tasks and models, unlike opaque metrics like perplexity.
  • Agent collaboration is powerful: The manager-worker agent setup allows for iterative refinement and diverse perspectives, leading to more robust criteria.
  • Knowledge base accelerates learning: Leveraging existing research in data quality significantly bootstraps the criterion mining process.
  • Small data, big impact: CritIQ demonstrates that a surprisingly small number of human annotations (~30 pairs) can be sufficient to mine high-quality criteria.

Key Results

  • High accuracy in criterion mining: CritIQ Flow achieved high accuracies on human-annotated test sets across code, math, and logic domains, outperforming baselines like TextGrad and single-criterion approaches like QuRating. For example, in the code domain, CritIQ Flow achieved 89.33% accuracy, compared to TextGrad's 72.70%.
  • Improved LLM performance: Continual pretraining of Llama 3.2 models on data selected by CritIQ Scorer resulted in improved performance on downstream tasks (code generation, math problem solving, logic reasoning) compared to models trained on uniformly sampled data. For instance, in code generation tasks, CritIQ-selected data led to higher scores on HumanEval/+ and MBPP/+.
  • Effectiveness of components: Ablation studies confirmed the benefits of the knowledge base and the iterative refinement process for improving criterion accuracy.
  • Majority voting is crucial: The paper showed that using majority voting among multiple criteria significantly improved accuracy compared to using a single criterion.

Limitations and Future Work

  • Domain specificity: The current experiments are limited to three specific domains (code, math, logic). Exploring general domain data selection remains an open challenge.
  • Human bias: While CritIQ reduces biases compared to handwritten criteria, human biases in the annotation process cannot be entirely eliminated, as defining "high quality" is inherently subjective.
  • Computational constraints: The current approach is limited to continual pretraining rather than pretraining from scratch, and uses a relatively modest model size (3B parameters) due to computational costs.
  • Future work: The authors suggest exploring scaling to larger models, more comprehensive training approaches, and investigating general domain data selection.

Summary updated

Abstract

Language model heavily depends on high-quality data for optimal performance. Existing approaches rely on manually designed heuristics, the perplexity of existing models, training classifiers, orcareful prompt engineering, which require significant expert experience and human annotation effort while introduce biases. We introduce CritiQ, a novel data selection method that automatically mines criteria from human preferences for data quality with only ~30 human-annotated pairs and performs efficient data selection. The main component, CritiQ Flow, employs a manager agent to evolve quality criteria and worker agents to make pairwise judgments. We build a knowledge base that extracts quality criteria from previous work to boost CritiQ Flow. Compared to perplexity- and classifier-based methods, verbal criteria are more interpretable and have greater reusable value. After deriving the criteria, we train the CritiQ Scorer to give quality scores and perform efficient data selection. We demonstrate the effectiveness of our method in the code, math, and logic domains, achieving high accuracy on human-annotated test sets. To validate the quality of the selected data, we continually train Llama 3.2 models and observe improved performance on downstream tasks compared to uniform sampling. Ablation studies validate the benefits of the knowledge base and the reflection process. We analyze how criteria evolve and the effectiveness of majority voting.
BibTeX
@inproceedings{guo2025critiq,
  author = {Honglin Guo and Kai Lv and Qipeng Guo and Tianyi Liang and Zhiheng Xi and Demin Song and Qiuyinzhe Zhang and Yu Sun and Kai Chen and Xipeng Qiu and Tao Gui},
  title = {{CritiQ: Mining Data Quality Criteria from Human Preferences}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure