CritiQ: Mining Data Quality Criteria from Human Preferences

Honglin Guo, Kai Lv, Qipeng Guo, Tianyi Liang, Zhiheng Xi, Demin Song, Qiuyinzhe Zhang, Yu Sun, Kai Chen, Xipeng Qiu, Tao Gui

TL;DR

CRITIQ is a novel framework that automatically mines interpretable data quality criteria from as few as 30 human-annotated pairwise comparisons. By using an iterative multi-agent workflow to evolve these criteria and training a lightweight scorer, CRITIQ enables high-accuracy data selection that significantly boosts LLM performance in specialized domains like code, math, and logic.

What Problem Does This Paper Address?

The performance of Large Language Models (LLMs) is inextricably linked to the quality of their training data. However, defining "high quality" is notoriously difficult and varies wildly across domains.

Existing data selection methods generally fall into four categories, each with significant drawbacks:

  1. Manual Heuristics: Rules like "length filtering" are brittle and require extensive expert effort.
  2. Perplexity-based: Using model loss as a proxy for quality often misses nuanced semantic value.
  3. Classifiers: These act as black boxes and require large amounts of labeled data.
  4. Prompt Engineering: Manually crafting "quality" prompts for LLMs is subjective and hard to generalize.

The authors identify a need for a method that captures nuanced human preferences with minimal manual effort, while remaining interpretable and adaptable to different domains.

Methods

CRITIQ introduces a two-stage process: CRITIQ Flow for mining criteria and the CRITIQ Scorer for large-scale execution.

1. CRITIQ Flow: The Agentic Evolution

Instead of asking a human to write a rubric, CRITIQ asks for ~30 pairwise comparisons (e.g., "Text A is better than Text B"). It then uses an agent workflow to figure out why:

  • Knowledge Base: The system starts with a library of 342 criteria extracted from existing research papers.
  • Manager Agent (GPT-4o): This agent selects initial criteria, generates new ones, and refines existing ones through reflection.
  • Worker Agents (Qwen2.5-72B): These agents apply specific criteria to the human-annotated pairs to see which ones align with human judgment.
  • Criteria Evolution: Criteria that perform poorly are discarded. Mid-performing criteria are sent back to the Manager for "reflection"—the Manager analyzes the "wrong cases" and refines the verbal description of the criterion to be more precise.
Figure 1: The overview of CRITIQ. We (1) employ hu- man annotators to annotate ∼30 pairwise quality com- parisons, (2) use CRITIQ Flow to mine quality criteria, (3) use the derived criteria to annotate 25k pairs, and (4) train the CRITIQ Scorer to perform efficient data selection.

2. CRITIQ Scorer: Scaling Up

Once the final set of high-accuracy criteria is mined, the Worker agents use them to label a much larger set (25,000 pairs). This data is used to train a lightweight CRITIQ Scorer (a Qwen2.5-1.5B model using a Bradley-Terry objective). This small model can then efficiently score millions of documents for final data selection.

Datasets

The authors evaluated CRITIQ across three distinct domains:

  • Code: Python subset of Stack v2.
  • Math: Non-code subset of OpenWebMath.
  • Logic: Zyda-2.

For evaluation, they used standard benchmarks including HumanEval/MBPP (Code), GSM8k/MATH (Math), and ARC-C/LogiQA (Logic).

Key Insights and Ideas

  • Verbal Criteria are Reusable: Unlike a black-box classifier, CRITIQ produces human-readable criteria (e.g., "algorithm efficiency," "logical rigor"). These can be audited by humans and reused across similar datasets.
  • The Power of Reflection: The iterative refinement process is highly effective. As shown in the accuracy distribution, criteria progressively concentrate toward higher alignment with human judgment over iterations.
  • Majority Voting Wins: The authors found that using multiple specific criteria and aggregating their votes is significantly more accurate than using a single, broad "quality" prompt.
  • Small Data, Big Impact: The fact that the system can bootstrap a high-performing scorer from only 30-40 human labels suggests that LLMs are already quite good at identifying quality if they are given the right verbal framework.
Figure 3: Evolution of distributions of the top-k Python code quality criteria through evolution iterations, where k is the number of the final criteria.

Key Results

CRITIQ demonstrated superior performance in both alignment and downstream model training:

  • Human Alignment: The CRITIQ Scorer achieved ~90% accuracy on human-annotated test sets, outperforming baselines like TextGrad and QuRating.
  • Model Improvement: The authors continually pre-trained a Llama-3.2-3B model on data selected by CRITIQ. In the Code domain, the model achieved an average score of 53.88% (HumanEval/MBPP), compared to 38.80% for the raw dataset and 47.93% for the next-best baseline (QuRating).
  • Efficiency: By training a 1.5B parameter scorer, the authors made the selection process computationally feasible for large-scale corpora while maintaining high accuracy.
Accuracies on the human-annotated Dtest. The best results and the best results without training a model are in bold.

Limitations and Future Work

  • Domain Specificity: While successful in technical domains (code, math), the authors acknowledge that general-domain data selection (e.g., creative writing or general web text) remains an open challenge.
  • Human Bias: The system relies on an initial seed of human preferences. If the initial 30 pairs are biased, the mined criteria will likely reflect those biases.
  • Model Scale: Due to compute constraints, the authors only tested continual pre-training on a 3B parameter model. Future work is needed to see how these selection benefits scale to 70B+ parameter models or training from scratch.

Summary updated

Abstract

Language model heavily depends on high-quality data for optimal performance. Existing approaches rely on manually designed heuristics, the perplexity of existing models, training classifiers, or careful prompt engineering, which require significant expert experience and human annotation effort while introduce biases. We introduce CritiQ, a novel data selection method that automatically mines criteria from human preferences for data quality with only ~30 human-annotated pairs and performs efficient data selection. The main component, CritiQ Flow, employs a manager agent to evolve quality criteria and worker agents to make pairwise judgments. We build a knowledge base that extracts quality criteria from previous work to boost CritiQ Flow. Compared to perplexity- and classifier- based methods, verbal criteria are more interpretable and possess reusable value. After deriving the criteria, we train the CritiQ Scorer to give quality scores and perform efficient data selection. We demonstrate the effectiveness of our method in the code, math, and logic domains, achieving high accuracy on human-annotated test sets. To validate the quality of the selected data, we continually train Llama 3.1 models and observe improved performance on downstream tasks compared to uniform sampling. Ablation studies validate the benefits of the knowledge base and the reflection process. We analyze how criteria evolve and the effectiveness of majority voting.
BibTeX
@inproceedings{guo2025critiq,
  author = {Honglin Guo and Kai Lv and Qipeng Guo and Tianyi Liang and Zhiheng Xi and Demin Song and Qiuyinzhe Zhang and Yu Sun and Kai Chen and Xipeng Qiu and Tao Gui},
  title = {{CritiQ: Mining Data Quality Criteria from Human Preferences}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure