Abstract
Improving training efficiency continues to be one of the primary challenges in large-scale Reinforcement Learning (RL). In this paper, we investigate how context length and the complexity of training data influence the RL scaling training process of R1-distilled reasoning models, e.g., DeepSeek-R1-Distill-Qwen-1.5B. Our experimental results reveal that: (1) simply controlling the context length and curating the training data based on the input prompt length can effectively improve the training efficiency of RL scaling, achieving better performance with more concise CoT; (2) properly scaling the context length helps mitigate entropy collapse; and (3) carefully choosing the context length facilitates achieving efficient LLM training and reasoning. Inspired by these insights, we propose FastCuRL, a curriculum RL framework with stage-wise context scaling to achieve efficient LLM training and reasoning. Extensive experimental results demonstrate that FastCuRL-1.5B-V3 significantly outperforms state-of-the-art reasoning models on five competition-level benchmarks and achieves 49.6% accuracy on AIME 2024. Furthermore, FastCuRL-1.5B-Preview surpasses DeepScaleR-1.5B-Preview on five benchmarks while only using a single node with 8 GPUs and a total of 50% of training steps.
Introduction
The recent success of models like DeepSeek-R1 has demonstrated that scaling Reinforcement Learning (RL) by allowing models to generate extensive Chain-of-Thought (CoT) sequences significantly enhances reasoning capabilities. However, this approach introduces two major challenges: computational waste due to excessively long CoT sequences and entropy collapse, where models become deterministic too quickly, leading to redundant thinking patterns and poor exploration. FastCuRL addresses these issues by introducing a curriculum-based approach to RL training that optimizes both context length and data complexity.
Methodology: FastCuRL Framework
FastCuRL (Fast Curriculum Reinforcement Learning) optimizes the Group Relative Policy Optimization (GRPO) algorithm through three key innovations:
Stage-wise Context Scaling
Instead of a linear increase in context length, FastCuRL employs a cyclical approach that alternates between CoT compression (long-to-short) and extension (short-to-long). This forces the model to refine its logic and prune redundant steps while maintaining the ability to handle complex problems.
Complexity-Aware Data Selection
The researchers discovered a strong correlation between input prompt length and the required reasoning length. Training data is stratified into three tiers (L1, L2, L3) based on prompt length, allowing the model to master concise logic on simpler problems before tackling more complex ones.
Entropy-Enhanced GRPO
An explicit entropy bonus is added to the objective function to act as "creativity insurance," preventing premature convergence and encouraging the model to navigate complex solution spaces effectively.
Experimental Setup
The study utilizes the DeepScaleR dataset, which includes approximately 40,000 unique math problems from AIME (1984-2023), AMC, Omni-MATH, and Still datasets. Models are evaluated on five competition-level benchmarks: MATH 500, AIME 2024, AMC 2023, Minerva Math, and OlympiadBench.
Key Insights and Analysis
A critical finding is that for a 1.5B model, a 16K token context window represents an "optimal" balance. Shorter windows lead to rapid entropy collapse, while excessively long windows (24K+) encourage redundant "thinking" without accuracy gains.
As shown in the analysis of complexity levels, longer prompts (L3) naturally lead to longer outputs and lower initial rewards, justifying the curriculum approach of starting with simpler, shorter prompts (L1) to establish a baseline of concise reasoning before scaling up.
Results
FastCuRL demonstrates exceptional efficiency and performance compared to existing baselines:
- Efficiency: FastCuRL-1.5B-Preview outperformed DeepScaleR-1.5B-Preview while using 50% fewer training steps and running on a single 8-GPU node.
- Performance: The final model, FastCuRL-1.5B-V3, achieved 49.6% on AIME 2024 and 90.5% on MATH 500.
- Conciseness: The models generate responses approximately 50% shorter than DeepSeek-R1-Distill-Qwen-1.5B while maintaining higher accuracy, proving that conciseness is a hallmark of quality reasoning.
Conclusion
FastCuRL proves that intelligent curriculum design and stage-wise context scaling can significantly reduce the computational cost of training high-performance reasoning models. While verified on a 1.5B parameter model, future work will explore whether these efficiency gains and context "sweet spots" scale linearly for larger models and more nuanced complexity metrics.
Summary updated