TL;DR
This paper argues that the structure of data, not just its size, is paramount for training Large Language Models (LLMs) in reasoning tasks. They introduce MCSQ, a framework to quantify data quality across six dimensions, and demonstrate that carefully curated data, even with "imperfect" samples, can outperform simply scaling up dataset size.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown impressive reasoning capabilities, often enhanced through supervised fine-tuning (SFT) on structured reasoning traces (question, reasoning trace, answer). However, the prevailing assumption has been that "more data is better." This paper questions this, highlighting that existing data quality evaluation methods are often heuristic and lack systematic frameworks. Practitioners lack clear guidance on how to best curate reasoning datasets, leading to potential inefficiencies and even performance degradation. The authors aim to provide a principled, quantitative approach to understanding and optimizing reasoning dataset structure for better LLM performance.
Methods
The core contribution is the MCSQ (Multi-dimensional Quantitative Framework for Evaluating Distilled Chain-of-Thought SFT Data Quality). MCSQ distills six key dimensions from Objective Information Theory (OIT) metrics, adapted for SFT data:
- Volume: Total number of samples.
- Scope: Breadth of subject domains covered.
- Granularity: Distribution of samples across domains.
- Variety: Diversity of reasoning types.
- Distortion: Degree of model disagreement on sample correctness.
- Mismatch: Relevance of samples to the intended reasoning task.
The authors systematically vary each of these dimensions in controlled experiments to isolate their impact on LLM reasoning performance and general capabilities. They use a Qwen2.5-3B-Instruct model for most experiments, fine-tuned using DeepSpeed ZeRO Stage 3.
Datasets
- s1-59k dataset: The primary dataset used for experiments, a large, high-quality collection of 59k (question, reasoning trace, answer) triplets. After preprocessing, they use s1-54k (54,046 samples). This dataset is notable for its multi-dimensional annotations (domain, reasoning type, model-based correctness/relevance), which are crucial for MCSQ.
- Reasoning-Intensive Tasks: AMC (2023), GPQA-diamond, MATH500, Minerva, and Olympiad-Bench.
- General Capability Tasks: MMLU, CMMLU, and C-Eval.
- DeepMath-103K: A challenging reasoning corpus used for additional validation.
Key Insights and Ideas
- Structure over Size: Dataset structure (Scope, Granularity, Variety, Distortion, Mismatch) consistently has a more significant impact on reasoning performance than sheer volume, especially beyond a moderate threshold.
- "Imperfect" Data Can Be Good: Selectively incorporating "distorted" (model-disagreed) or "mismatched" (low-relevance) samples can surprisingly boost performance on certain advanced reasoning benchmarks, challenging the notion that only perfectly "clean" data is beneficial.
- Trade-off Between Specialization and Generalization: Fine-tuning for reasoning skills can lead to a decline in general knowledge capabilities (catastrophic forgetting). This highlights the need for data-efficient and structure-aware curation.
- Optimal Scope is Task-Dependent: Intermediate domain scope settings often outperform both very narrow and very broad coverage.
- Automated Relevance Filters Can Be Overly Conservative: Discarding data labeled "low relevance" by automated scoring might remove valuable edge-case samples.
Key Results
- Volume: Performance leaps from 100 to 1,000 samples, but gains diminish significantly beyond 10,000 samples. Fine-tuning on 54k samples yields high reasoning scores but degrades general ability by 2-4 points.
- Scope: Intermediate scope settings (20-30 domains) often perform best on Pass@1, while broader scope generally helps Pass@5 and general capability.
- Granularity: Finer granularity (more uniform distribution) enhances reasoning performance but can degrade generalizability. A balanced trade-off is achieved with G2 (uniform sampling) and G1 (original distribution).
- Variety: Maximizing reasoning type diversity offers only marginal overall gains but can improve solution diversity and model robustness.
- Distortion: High-distortion data (D3) yields the best Pass@5 on some benchmarks, outperforming "pure" data. However, excessive noise can harm the model.
- Mismatch: Data labeled "low relevance" by automated scoring can significantly boost Pass@5 on GPQA-Diamond.
Limitations and Future Work
- Model Dependency: Experiments were primarily conducted on the Qwen family; validation on other architectures is needed.
- Inherent Correlations: Isolating MCSQ dimensions perfectly is challenging due to real-world data correlations; their control is approximate.
- Metric Operationalization: Distortion and Mismatch rely on model judgments, which could be refined.
- Specialization-Generalization Trade-off: Further investigation is needed to understand and mitigate this trade-off.
- Computational Constraints: Limited the number of configurations and hyperparameter tuning.
The authors suggest future work on extending MCSQ to multilingual/coding tasks, exploring more dynamic/multi-turn SFT scenarios, and developing strategies like a "three-stage filtering funnel" to balance reasoning specialization and general capabilities.
Summary updated