Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs). However, training on high-difficulty tasks often faces a "cold-start" problem where the model's base success rate is too low to provide meaningful reward signals. We introduce QuestA (Question Augmentation), a curriculum-based scaffolding strategy that injects partial solutions as hints during training. This approach enables a 1.5B-parameter model to achieve state-of-the-art results on elite mathematics benchmarks, including AIME and HMMT, often surpassing models with over 20 times its parameter count.
Introduction
The development of reasoning models like DeepSeek-R1 and OpenAI’s o1 relies heavily on RLVR. A critical challenge in this paradigm is the "Goldilocks" problem: easy problems lead to entropy collapse and overfitting, while extremely difficult problems (e.g., AIME-level math) offer no positive reward signals because the model's initial success rate is near zero. QuestA addresses this by providing a bridge between the model's current capabilities and the complexity of target tasks.
Methodology: Question Augmentation (QuestA)
QuestA is an input-level strategy that modifies training prompts to facilitate learning on unreachable problems. The core components include:
- Partial Solution Injection: For the most challenging problems, the first $p\%$ of a high-quality solution sketch is prepended as a "Hint" in the prompt.
- Curriculum Learning: A two-stage curriculum is employed. Initially, generous hints (e.g., 50% of the solution) make problems tractable. As the model improves, the hint size is reduced (e.g., to 25%), increasing the reasoning burden on the model.
- RL Framework: The models are trained using Group Relative Policy Optimization (GRPO) without the standard KL-divergence penalty, prioritizing verifiable rewards such as correctness and formatting.
- Variance-Based Filtering: The method targets "high-variance" problems—those where the model occasionally succeeds—to maximize the strength of the learning signal.
Experimental Setup
We utilized the OpenR1-Math-220K dataset, filtering it to the 26,000 hardest problems where the base model, Nemotron-1.5B, initially failed. Evaluation was conducted across several elite math benchmarks: AIME24, AIME25, HMMT Feb 2025, Olympiad Bench, and BRUMO25. To assess generalization, we also tested the model on GPQA Diamond (science), Zebralogic (logic), and coding benchmarks like Codeforces and LCB.
Key Results and Analysis
QuestA-Nemotron-1.5B demonstrates remarkable performance for its size. As shown in Figure 1, the 1.5B model achieves 72.50% on AIME24 and 62.29% on AIME25, matching or outperforming the 32B DeepSeek-R1-Distill model across multiple benchmarks.
Scaling and Generalization
Analysis of Pass@k accuracy (Figure 2) reveals that QuestA-trained models maintain superior performance across various sample sizes. Theoretically, hints increase reward density by reducing the number of consecutive difficult steps the model must solve independently. If a solution requires two steps each with probability $\delta$, the total success probability $\delta^2$ is negligible. A hint for the first step restores the success probability to $\delta$, making training feasible within a reasonable sampling budget.
Crucially, training with hints does not create a dependency. The model's "capacity set" expands, allowing it to solve these difficult problems without hints at test time. Furthermore, the model showed increased entropy during training, suggesting that the scaffolding encourages the exploration of diverse reasoning paths.
Conclusion
QuestA provides a simple yet effective mechanism to scaffold LLMs toward mastering complex reasoning tasks. By bridging the gap in RL reward signals, a 1.5B model can achieve performance levels previously reserved for much larger architectures. Future work will explore bootstrapping hints in domains without existing solution sketches and refining hint selection to target pivotal reasoning steps.
Summary updated