Abstract
FastCuRL is a curriculum-based reinforcement learning (RL) framework designed to optimize the training of reasoning models like DeepSeek-R1. By implementing a stage-wise context scaling strategy that alternates between context compression and extension, FastCuRL achieves state-of-the-art performance on mathematical benchmarks while significantly reducing computational overhead. Specifically, the FastCuRL-1.5B-V3 model reached 49.6% accuracy on AIME 2024 using 50% fewer training steps than previous methods, demonstrating that strategic curriculum design can lead to both higher efficiency and superior reasoning capabilities in small-scale language models.
Introduction
The development of reasoning models, exemplified by DeepSeek-R1, has highlighted the power of scaling Reinforcement Learning to foster extended Chain-of-Thought (CoT) processes. However, this scaling introduces significant challenges: high computational costs due to long CoT generation, "overthinking" where models generate redundant or irrelevant content, and entropy collapse leading to premature convergence. Existing approaches often struggle with these issues, resulting in inefficient learning and sub-optimal final performance. FastCuRL addresses these hurdles by treating context length as a dynamic training variable.
Methodology: Stage-wise Context Scaling
The core of FastCuRL is a curriculum RL approach termed Stage-wise Context Scaling. This method moves away from fixed context limits, instead employing a "Compress-Extend Cycle."
The Compress-Extend Cycle
Rather than a linear progression from short to long context, FastCuRL strategically alternates context windows. By forcing the model to be concise (compression) and then allowing deeper exploration (extension), the framework encourages the model to prune redundant reasoning steps while maintaining logical depth. This iterative refinement helps the model avoid repetitive thinking patterns.
Complexity-Aware Data Selection
The researchers observed a strong correlation between the length of an input prompt and the reasoning depth required to solve it. To leverage this, training data was stratified into tiers based on prompt length.
This stratification allows the training process to match problem difficulty with the model's current context window, ensuring a smooth learning curve and preventing the model from being overwhelmed by complex problems too early.
Algorithmic Enhancements
FastCuRL utilizes a modified version of Group Relative Policy Optimization (GRPO). To combat entropy collapse—where a model becomes "too certain" of a sub-optimal path—an explicit entropy bonus was added to the objective function, maintaining the model's exploratory drive throughout the training stages.
Key Results and Discussion
FastCuRL demonstrates substantial improvements in both performance and efficiency. The training process, divided into five distinct stages (S-1 to S-5), shows a clear upward trajectory in accuracy as context and complexity are scaled.
Performance Benchmarks
The flagship FastCuRL-1.5B-V3 achieved a 49.6% accuracy on the AIME 2024 benchmark, surpassing much larger models and setting a new standard for 1.5B parameter reasoning models. Even the "Preview" version outperformed DeepScaleR-1.5B-Preview across multiple benchmarks while requiring only half the training steps and running on a single 8-GPU node.
Efficiency and Conciseness
A notable outcome of the FastCuRL framework is the reduction in model verbosity. On AIME 2024, the FastCuRL-1.5B-Preview model generated responses with an average of 8,654 tokens, nearly 50% fewer than the 16,390 tokens produced by the standard DeepSeek-R1-Distill-Qwen-1.5B. This indicates that the model has learned to reason more effectively without unnecessary "overthinking," proving that "more thinking" is not always "better thinking."
Conclusion
FastCuRL proves that curriculum-based context management is a powerful tool for training efficient reasoning models. By identifying a "sweet spot" for context length and using input length as a proxy for complexity, the framework achieves high performance with minimal hardware. Future work will explore scaling these strategies to larger model sizes and developing more adaptive mechanisms for context and penalty adjustment.
Summary updated