The recent success of "thinking" models like DeepSeek-R1 has demonstrated that Reinforcement Learning (RL) can induce emergent reasoning capabilities in Large Language Models (LLMs). However, applying RL to complex reasoning tasks faces a fundamental "cold start" problem: if a model cannot solve a problem even once through random exploration, it receives no reward signal to learn from. Furthermore, the search space for a correct reasoning chain grows exponentially with the number of steps required, leading to the "sparse reward" problem.
To overcome these barriers, researchers have converged on Curriculum Reinforcement Learning, a strategy that systematically increases the difficulty and complexity of tasks to guide the model from basic logic to competition-level problem-solving.
Solving the Cold Start with Horizon Curricula
The most intuitive curriculum is based on the length of the reasoning chain, often called the "horizon." In h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning, Alesia Ivanova and colleagues demonstrate that by starting with single-step math problems and incrementally adding complexity, the sample complexity of RL drops from exponential to polynomial.
This "stagewise horizon curriculum" prevents the model from becoming overwhelmed. Similarly, Leonardo Ranaldi and the team behind R2-MultiOmnia found that gradually increasing the allowable token length for reasoning chains prevents models from developing "degenerate" behaviors, such as repetitive or hallucinatory outputs, which often occur when a model is forced to "think" longer than its current logical capacity allows.
Stabilizing Training via Task Complexity
Beyond length, the nature of the task provides a vital lever for stability. In the medical domain, Shengjie Rui introduced MedCCO, which trains first on discrete multiple-choice questions before moving to complex free-text reasoning. This allows the model to establish a stable foundation using "sharp" binary rewards (correct/incorrect) before adapting to the high-variance rewards associated with semantic reasoning.
A similar pattern appears in multimodal reasoning. Ruifeng Yuan (PCuRL) and Ke Niu (CReFT-CAD) both utilize curricula that progress from simple verification tasks (e.g., "Is this 2D drawing correct?") to generative reasoning (e.g., "Extract the 3D parameters from this drawing"). This structured progression ensures the model doesn't collapse when faced with the high-dimensional search space of geometric or visual logic.
Nudging the Boundaries of the Unsolvable
What happens when a model plateaus? Standard RL often fails to push a model past its current "ceiling." Becky Xiangyu Peng and the creators of NuRL address this by adaptively injecting self-generated, abstract hints into the training process. These "nudges" provide a learning signal for previously unsolvable problems, effectively raising the model's reasoning ceiling.
Another approach to bridging this gap is BREAD, proposed by Xuechen Zhang. BREAD uses "expert anchors"—prefixes of correct reasoning chains—and gradually shortens them. By forcing the model to solve only the final steps of a problem first, and then progressively requiring it to initiate the reasoning itself, the framework creates a natural curriculum that speeds up training by 3x compared to standard RL.
Math as the Foundation for Broad Reasoning
A recurring theme across this research is the role of mathematics as a "bootstrapping" domain. Deqian Kong and researchers at Salesforce AI found that training a model on math-only RL first, and then expanding to other subjects, allows even small 4B-parameter models to outperform much larger 32B-parameter giants in logic and STEM.
This suggests that mathematical reasoning provides a clean, verifiable environment for a model to learn the mechanics of thinking—such as self-correction and verification—which then generalize to more ambiguous domains. Yiyao Yu (CoR) extends this by sequencing reasoning paradigms: starting with natural language, moving to algorithmic code, and finally mastering formal symbolic proofs.
Mitigating Reward Hacking and Collapse
Curriculum learning isn't just about performance; it's about safety and stability. Fahim Tajwar warns that prolonged self-training can lead to "reward hacking," where a model collapses into a single "template" answer to maximize internal rewards. A progressive curriculum of increasing difficulty acts as a safeguard, ensuring the model maintains logical consistency on simpler tasks before being exposed to the noisier signals of complex, self-rewarded exploration.
By treating reasoning not as a single skill to be "unlocked," but as a ladder of increasingly complex behaviors—from retrieval to tracing to synthesis—these curriculum-based approaches allow models to "pull themselves up by their bootstraps" and reach performance levels previously reserved for much larger, human-annotated systems.