TL;DR
CRITIQ is a novel framework that automatically mines interpretable data quality criteria from as few as 30 human-annotated pairwise comparisons. By using an iterative multi-agent workflow to evolve these criteria and training a lightweight scorer, CRITIQ enables high-accuracy data selection that significantly boosts LLM performance in specialized domains like code, math, and logic.
What Problem Does This Paper Address?
The performance of Large Language Models (LLMs) is inextricably linked to the quality of their training data. However, defining "high quality" is notoriously difficult and varies wildly across domains.
Existing data selection methods generally fall into four categories, each with significant drawbacks:
- Manual Heuristics: Rules like "length filtering" are brittle and require extensive expert effort.
- Perplexity-based: Using model loss as a proxy for quality often misses nuanced semantic value.
- Classifiers: These act as black boxes and require large amounts of labeled data.
- Prompt Engineering: Manually crafting "quality" prompts for LLMs is subjective and hard to generalize.
The authors identify a need for a method that captures nuanced human preferences with minimal manual effort, while remaining interpretable and adaptable to different domains.
Methods
CRITIQ introduces a two-stage process: CRITIQ Flow for mining criteria and the CRITIQ Scorer for large-scale execution.
1. CRITIQ Flow: The Agentic Evolution
Instead of asking a human to write a rubric, CRITIQ asks for ~30 pairwise comparisons (e.g., "Text A is better than Text B"). It then uses an agent workflow to figure out why:
- Knowledge Base: The system starts with a library of 342 criteria extracted from existing research papers.
- Manager Agent (GPT-4o): This agent selects initial criteria, generates new ones, and refines existing ones through reflection.
- Worker Agents (Qwen2.5-72B): These agents apply specific criteria to the human-annotated pairs to see which ones align with human judgment.
- Criteria Evolution: Criteria that perform poorly are discarded. Mid-performing criteria are sent back to the Manager for "reflection"—the Manager analyzes the "wrong cases" and refines the verbal description of the criterion to be more precise.
2. CRITIQ Scorer: Scaling Up
Once the final set of high-accuracy criteria is mined, the Worker agents use them to label a much larger set (25,000 pairs). This data is used to train a lightweight CRITIQ Scorer (a Qwen2.5-1.5B model using a Bradley-Terry objective). This small model can then efficiently score millions of documents for final data selection.
Datasets
The authors evaluated CRITIQ across three distinct domains:
- Code: Python subset of Stack v2.
- Math: Non-code subset of OpenWebMath.
- Logic: Zyda-2.
For evaluation, they used standard benchmarks including HumanEval/MBPP (Code), GSM8k/MATH (Math), and ARC-C/LogiQA (Logic).
Key Insights and Ideas
- Verbal Criteria are Reusable: Unlike a black-box classifier, CRITIQ produces human-readable criteria (e.g., "algorithm efficiency," "logical rigor"). These can be audited by humans and reused across similar datasets.
- The Power of Reflection: The iterative refinement process is highly effective. As shown in the accuracy distribution, criteria progressively concentrate toward higher alignment with human judgment over iterations.
- Majority Voting Wins: The authors found that using multiple specific criteria and aggregating their votes is significantly more accurate than using a single, broad "quality" prompt.
- Small Data, Big Impact: The fact that the system can bootstrap a high-performing scorer from only 30-40 human labels suggests that LLMs are already quite good at identifying quality if they are given the right verbal framework.
Key Results
CRITIQ demonstrated superior performance in both alignment and downstream model training:
- Human Alignment: The CRITIQ Scorer achieved ~90% accuracy on human-annotated test sets, outperforming baselines like TextGrad and QuRating.
- Model Improvement: The authors continually pre-trained a Llama-3.2-3B model on data selected by CRITIQ. In the Code domain, the model achieved an average score of 53.88% (HumanEval/MBPP), compared to 38.80% for the raw dataset and 47.93% for the next-best baseline (QuRating).
- Efficiency: By training a 1.5B parameter scorer, the authors made the selection process computationally feasible for large-scale corpora while maintaining high accuracy.
Limitations and Future Work
- Domain Specificity: While successful in technical domains (code, math), the authors acknowledge that general-domain data selection (e.g., creative writing or general web text) remains an open challenge.
- Human Bias: The system relies on an initial seed of human preferences. If the initial 30 pairs are biased, the mined criteria will likely reflect those biases.
- Model Scale: Due to compute constraints, the authors only tested continual pre-training on a 3B parameter model. Future work is needed to see how these selection benefits scale to 70B+ parameter models or training from scratch.
Summary updated