Unleashing the Reasoning Potential of LLMs by Critique Fine-Tuning on One Problem

Yubo Wang, Ping Nie, Kai Zou, Lijun Wu, Wenhu ChenEMNLP 2025

TL;DR

This paper introduces "Critique Fine-Tuning" (CFT), a novel and highly efficient method to unlock the reasoning capabilities of Large Language Models (LLMs). By fine-tuning LLMs on critiques of a single problem's solutions, CFT achieves performance comparable to or even surpassing more computationally expensive methods like Reinforcement Learning (RL), using a fraction of the compute and data.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown immense potential in reasoning tasks, but fully unleashing this potential often requires extensive post-training methods like Reinforcement Learning (RL). While RL can be effective, it's notoriously expensive (hundreds of GPU hours) and unstable. Supervised Fine-Tuning (SFT) is more stable but demands large, high-quality datasets, which are scarce for many reasoning tasks. The core problem this paper addresses is: Can we achieve significant reasoning improvements in LLMs with a more efficient, data-light, and compute-friendly approach? The authors propose that by focusing on critiquing incorrect solutions, rather than just generating correct ones, we can effectively teach LLMs to reason better.

Methods

The paper introduces Critique Fine-Tuning (CFT), a method that leverages a "teacher-student" paradigm. Here's how it works:

  1. Dataset Construction (One-Shot CFT):
    • A single "seed problem" is chosen.
  2. Multiple LLMs (solution generators) are used to produce a diverse set of candidate solutions for this problem.
  3. Stronger "teacher LLMs" then provide detailed critiques for each candidate solution, identifying errors and explaining the correct approach.
  4. This process creates a compact dataset of (problem, candidate solution, critique) triplets.
  5. Fine-Tuning:
    • The LLM being trained (the "student") is fine-tuned on this dataset.
  6. The input to the model is the problem and a candidate solution, and the target output is the teacher's critique.
  7. This teaches the model to analyze solutions, identify flaws, and understand the reasoning process, rather than just memorizing correct answers.

This approach is contrasted with traditional SFT, which trains models to directly generate correct solutions based on prompts and reference answers.

Datasets

The authors use existing math and logic reasoning benchmarks for evaluation.

  • Math Reasoning: They select four representative problems from the DeepScaleR subset for their seed problems. Evaluation is performed on six math benchmarks, including Math-500, Minerva, Olympiad, AIME24, AIME25, and AMC23.
  • Logic Reasoning: They evaluate on three subtasks from the BIG-Bench Extra Hard (BBEH) benchmark: Causal Understanding, DisambiguationQA, and Time Arithmetic.

No new benchmarks are introduced, but the paper focuses on creating small, targeted datasets for CFT from single problems.

Key Insights and Ideas

  • Critique is Powerful: The core insight is that learning to critique and identify errors is a highly effective way to improve reasoning, potentially more so than simply imitating correct solutions.
  • One-Shot Learning for Reasoning: The paper demonstrates that even a single problem's critique data can be sufficient to "unleash" significant reasoning potential in LLMs. This dramatically reduces data requirements.
  • Compute Efficiency: CFT is orders of magnitude more compute-efficient than RLVR, achieving comparable or better results with significantly less GPU time (e.g., 5 GPU hours vs. 120+ GPU hours).
  • Robustness to Seed Problems: While some seed problems might be more effective (moderately difficult ones), CFT is generally robust across different seed choices.
  • Diversity Matters: Using diverse candidate solutions generated by multiple models leads to richer critiques and better CFT performance.

Key Results

  • Significant Performance Gains: CFT consistently improves reasoning accuracy across various LLM scales (1.5B to 14B parameters) and tasks.
  • Math Reasoning: Qwen2.5-Math-7B-CFT achieved an average improvement of 15% on six math benchmarks and 16% on three logic reasoning benchmarks, using only 5 GPU hours.
  • Outperforming SFT: CFT substantially outperforms SFT, even when SFT is trained on a much larger dataset (full DeepScaleR dataset vs. one-shot CFT data). For example, on Qwen2.5-Math-7B, one-shot CFT achieved 42.2% accuracy compared to 25.6% for full SFT.
  • Competing with RLVR: CFT matches or surpasses one-shot RLVR in most settings, with significantly less compute. For instance, on Qwen2.5-Math-7B, CFT outperformed RLVR by +2.0 points.
  • Logic Reasoning: CFT showed strong in-domain and cross-task generalization on BBEH subtasks, outperforming SFT consistently.
  • Model Scale: CFT's benefits are more pronounced at larger model scales, demonstrating its ability to unlock latent capabilities.

Limitations and Future Work

  • Model Scale Limitation: The study primarily focused on relatively weaker base LLMs. The authors suggest that while CFT is effective at unleashing latent capabilities, its impact might be less pronounced on already heavily aligned or stronger LLMs. Future work could explore adaptations for more capable models.
  • Potential Misuse: Enhanced reasoning capabilities could be misused for generating plausible but incorrect mathematical content or for academic dishonesty. The authors encourage responsible use and further research into safeguards.
  • Future Directions: Investigating adaptations of CFT for more capable or well-aligned models is a key future direction.

Summary updated

BibTeX
@inproceedings{wang2025unleashing,
  author = {Yubo Wang and Ping Nie and Kai Zou and Lijun Wu and Wenhu Chen},
  title = {{Unleashing the Reasoning Potential of LLMs by Critique Fine-Tuning on One Problem}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure