Abstract
The release of DeepSeek-R1 marked a paradigm shift in the development of large language models, demonstrating that explicit, long-form reasoning can be elicited through Reinforcement Learning from Verifiable Rewards (RLVR). While the model weights were made public, the specific training recipes remained proprietary. This report synthesizes 100 days of intense open-source replication efforts, detailing the methodologies of Supervised Fine-Tuning (SFT) and RLVR. We explore data construction, algorithmic design, and key findings from the research community to provide a comprehensive roadmap for the next generation of Reasoning Language Models (RLMs).
Introduction
DeepSeek-R1 fundamentally changed the AI landscape by achieving state-of-the-art reasoning performance at a fraction of the training cost of proprietary competitors. Its success sparked a "Cambrian explosion" of replication studies aimed at reproducing "Aha moments"—emergent self-correction and reflection—in smaller, open-source models. This survey bridges the gap between high-level technical reports and the practical implementation details discovered by the community, focusing on how researchers can bridge the performance gap through structured training and curated data.
Training Methodologies: SFT and RLVR
Replication efforts generally follow a two-stage training pipeline designed to instill and then refine reasoning capabilities.
1. Supervised Fine-Tuning (SFT)
Most replications begin by "distilling" reasoning traces from DeepSeek-R1. By training a smaller base model (such as Qwen-7B) on the long-form thought processes generated by R1, the model learns the necessary format and style of chain-of-thought (CoT) reasoning. Research indicates that quality is paramount; a small "cold-start" dataset of high-quality reasoning chains is often superior to massive, noisy datasets.
2. Reinforcement Learning from Verifiable Rewards (RLVR)
RLVR is the core innovation of the R1 paradigm. Unlike standard RLHF, which relies on subjective human preferences, RLVR utilizes objective, rule-based rewards. For mathematical problems, the reward is binary based on the correctness of the final answer; for coding, it is based on passing unit tests.
While PPO remains a standard, Group Relative Policy Optimization (GRPO) has emerged as the preferred algorithm for the open-source community. GRPO is significantly more memory-efficient as it eliminates the need for a separate "critic" model, instead calculating rewards relative to a group of outputs for the same prompt. This lowers the hardware barrier, allowing more researchers to participate in RLM development.
The Evolving Data Landscape
The community has rapidly developed a complex ecosystem of datasets to support both SFT and RLVR stages. Key SFT datasets include OpenThoughts-114k, OpenR1-Math, and LIMO, the latter of which demonstrated that as few as 817 extremely high-quality samples can kickstart reasoning.
For RL training, massive collections of verifiable problems are required. Projects like DeepScaleR (40k problems), Skywork-OR1 (129k), and Math-RLVR (773k) provide the scale necessary for models to explore and refine their reasoning paths. Performance is typically measured against rigorous benchmarks like AIME24/25 and MATH500.
Key Insights and Performance Milestones
- Reproducible Emergence: Multiple studies confirm that RLVR alone can induce self-correction and reflection without human-labeled traces.
- Small Model Efficiency: Replicated models like DeepScaleR-1.5B have surpassed much larger models (including the original o1-preview) on AIME24 by scaling RL training and test-time compute.
- Performance Parity: Skywork-OR1-32B achieved an 80% score on AIME24, proving that open-source replications can reach the same tier as top-tier proprietary models.
- Addressing Length Bias: To prevent models from "cheating" by writing longer responses to appear more thorough, new variants like Dr. GRPO introduce penalties for unnecessary verbosity.
Challenges and Future Frontiers
Despite rapid progress, several hurdles remain. Current RLVR success is heavily dependent on "verifiable" rewards, making it difficult to apply to "soft" domains like creative writing or ethics where no single correct answer exists. Additionally, models occasionally suffer from "overthinking" simple queries or bypassing safety filters as their reasoning capabilities improve.
The next frontier involves moving beyond text-based reasoning toward Multimodal RLVR, where models must apply similar verifiable feedback loops to reason through images, diagrams, and video content.
Summary updated