100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models

Chong Zhang, Yue Deng, Xiang Lin, Bin Wang, Dianwen Ng, Hai Ye, Xingxuan Li, Yao Xiao, Zhanfeng Mo, Qi Zhang, Lidong BingarXiv 2025 · OpenReview

Abstract

The release of DeepSeek-R1 marked a paradigm shift in the development of large language models, demonstrating that explicit, long-form reasoning can be elicited through Reinforcement Learning from Verifiable Rewards (RLVR). While the model weights were made public, the specific training recipes remained proprietary. This report synthesizes 100 days of intense open-source replication efforts, detailing the methodologies of Supervised Fine-Tuning (SFT) and RLVR. We explore data construction, algorithmic design, and key findings from the research community to provide a comprehensive roadmap for the next generation of Reasoning Language Models (RLMs).

Introduction

DeepSeek-R1 fundamentally changed the AI landscape by achieving state-of-the-art reasoning performance at a fraction of the training cost of proprietary competitors. Its success sparked a "Cambrian explosion" of replication studies aimed at reproducing "Aha moments"—emergent self-correction and reflection—in smaller, open-source models. This survey bridges the gap between high-level technical reports and the practical implementation details discovered by the community, focusing on how researchers can bridge the performance gap through structured training and curated data.

Training Methodologies: SFT and RLVR

Replication efforts generally follow a two-stage training pipeline designed to instill and then refine reasoning capabilities.

1. Supervised Fine-Tuning (SFT)

Most replications begin by "distilling" reasoning traces from DeepSeek-R1. By training a smaller base model (such as Qwen-7B) on the long-form thought processes generated by R1, the model learns the necessary format and style of chain-of-thought (CoT) reasoning. Research indicates that quality is paramount; a small "cold-start" dataset of high-quality reasoning chains is often superior to massive, noisy datasets.

2. Reinforcement Learning from Verifiable Rewards (RLVR)

RLVR is the core innovation of the R1 paradigm. Unlike standard RLHF, which relies on subjective human preferences, RLVR utilizes objective, rule-based rewards. For mathematical problems, the reward is binary based on the correctness of the final answer; for coding, it is based on passing unit tests.

Taxonomy of training techniques for Reasoning Language Models (RLMs).

While PPO remains a standard, Group Relative Policy Optimization (GRPO) has emerged as the preferred algorithm for the open-source community. GRPO is significantly more memory-efficient as it eliminates the need for a separate "critic" model, instead calculating rewards relative to a group of outputs for the same prompt. This lowers the hardware barrier, allowing more researchers to participate in RLM development.

The Evolving Data Landscape

The community has rapidly developed a complex ecosystem of datasets to support both SFT and RLVR stages. Key SFT datasets include OpenThoughts-114k, OpenR1-Math, and LIMO, the latter of which demonstrated that as few as 817 extremely high-quality samples can kickstart reasoning.

Lineage and relationships between major open-source reasoning datasets.

For RL training, massive collections of verifiable problems are required. Projects like DeepScaleR (40k problems), Skywork-OR1 (129k), and Math-RLVR (773k) provide the scale necessary for models to explore and refine their reasoning paths. Performance is typically measured against rigorous benchmarks like AIME24/25 and MATH500.

Key Insights and Performance Milestones

  • Reproducible Emergence: Multiple studies confirm that RLVR alone can induce self-correction and reflection without human-labeled traces.
  • Small Model Efficiency: Replicated models like DeepScaleR-1.5B have surpassed much larger models (including the original o1-preview) on AIME24 by scaling RL training and test-time compute.
  • Performance Parity: Skywork-OR1-32B achieved an 80% score on AIME24, proving that open-source replications can reach the same tier as top-tier proprietary models.
  • Addressing Length Bias: To prevent models from "cheating" by writing longer responses to appear more thorough, new variants like Dr. GRPO introduce penalties for unnecessary verbosity.

Challenges and Future Frontiers

Despite rapid progress, several hurdles remain. Current RLVR success is heavily dependent on "verifiable" rewards, making it difficult to apply to "soft" domains like creative writing or ethics where no single correct answer exists. Additionally, models occasionally suffer from "overthinking" simple queries or bypassing safety filters as their reasoning capabilities improve.

The next frontier involves moving beyond text-based reasoning toward Multimodal RLVR, where models must apply similar verifiable feedback loops to reason through images, diagrams, and video content.

Summary updated

Abstract

The recent development of reasoning language models (RLMs) represents a novel evolution in large language models. In particular, the recent release of DeepSeek-R1 has generated widespread social impact and sparked enthusiasm in the research community for exploring the explicit reasoning paradigm of language models. However, the implementation details of the released models have not been fully open-sourced by DeepSeek, including DeepSeek-R1-Zero, DeepSeek-R1, and the distilled small models. As a result, many replication studies have emerged aiming to reproduce the strong performance achieved by DeepSeek-R1, reaching comparable performance through similar training procedures and fully open-source data resources. These works have investigated feasible strategies for supervised fine-tuning (SFT) and reinforcement learning from verifiable rewards (RLVR), focusing on data preparation and method design, yielding various valuable insights. In this report, we provide a summary of recent replication studies to inspire future research. We primarily focus on SFT and RLVR as two main directions, introducing the details for data construction, method design and training procedure of current replication studies. Moreover, we conclude key findings from the implementation details and experimental results reported by these studies, anticipating to inspire future research. We also discuss additional techniques of enhancing RLMs, highlighting the potential of expanding the application scope of these models, and discussing the challenges in development. By this survey, we aim to help researchers and developers of RLMs stay updated with the latest advancements, and seek to inspire new ideas to further enhance RLMs.
BibTeX
@misc{zhang2025days,
  author = {Chong Zhang and Yue Deng and Xiang Lin and Bin Wang and Dianwen Ng and Hai Ye and Xingxuan Li and Yao Xiao and Zhanfeng Mo and Qi Zhang and Lidong Bing},
  title = {{100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models}},
  year = {2025},
  eprint = {2505.00551},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.00551},
}
Figure