TL;DR
STONE (Self-Taught frOm Nearby Easier problems) is a label-free self-distillation method that allows language models to improve their reasoning without any ground-truth solutions. By generating and consistently solving simpler "stepping stone" problems, the model creates a high-quality internal teacher that outperforms traditional self-distillation—even when those traditional methods have access to the correct answers.
What Problem Does This Paper Address?
Training Large Language Models (LLMs) to reason through complex math and logic typically requires massive amounts of high-quality, human-labeled "chain-of-thought" data. On-policy self-distillation (OPSD) attempts to bypass this by letting a model learn from its own attempts, but it still usually requires a ground-truth answer key to "score" which attempts are worth learning from.
The obvious label-free alternative is self-consistency (voting): if the model generates the same answer multiple times, it’s probably right. However, voting fails exactly where we need it most: on hard problems. On difficult tasks, the model’s attempts often disagree, and the most frequent answer is frequently wrong. The authors address this "consensus gap" by asking: If a model can't agree on a hard problem, can it at least agree on easier versions of that problem?
Methods
The authors introduce STONE, which operates in three distinct steps:
- Find Uncertain Problems: The model identifies training problems where it lacks consensus (i.e., its attempts result in many different answers).
- Pose and Verify "Stepping Stones": For these hard problems, the model is prompted to write several easier related problems (e.g., a sub-problem, a special case with smaller numbers, or a simplified version). The model then attempts to solve these "stones" multiple times. It only keeps stones where it reaches a high level of internal agreement (consensus).
- Distill with Stones: The model acts as a "teacher" by looking at the worked solutions of these verified stepping stones. While the teacher sees this extra context, the "student" (the same model being trained) sees only the original hard problem. The student is trained to match the teacher's token-level probabilities, effectively "distilling" the reasoning logic derived from the easier problems into the original hard task.
Crucially, no ground-truth labels are ever read during training. The model relies entirely on its own ability to find consistency in simpler domains.
Datasets
- Training: 29,434 math problems from the OPSD training set (sourced from OpenThoughts).
- Evaluation: The models were tested across five rigorous math benchmarks: AIME24, AIME25, HMMT25, MATH500, and AMC23.
Key Insights and Ideas
- Agreement as a Proxy for Truth: While agreement is a poor indicator of correctness on hard problems, it is a very strong indicator on easier ones. STONE moves the "agreement requirement" to a complexity level where the model is actually reliable.
- The Teacher Doesn't Need the Answer: Surprisingly, the teacher model doesn't need to know the answer to the target problem to be effective. Providing worked solutions to related problems provides enough reasoning context to help the teacher guide the student's path.
- Worked Examples > Final Answers: In "non-thinking" mode (where models don't use long reasoning traces), STONE significantly outperformed OPSD with ground-truth labels. This suggests that worked examples of related problems are a more effective "teaching" signal than just knowing the final answer to the target problem.
Key Results
The researchers tested STONE on Qwen3-4B, Qwen3-8B, and Gemma 4 E4B in both "thinking" (long reasoning) and "non-thinking" modes.
- Massive Gains in Non-Thinking Mode: For Qwen3-8B, STONE improved average accuracy by 22.7 points, compared to only 8.5 points for OPSD with ground-truth labels.
- State-of-the-Art Label-Free Performance: In thinking mode, STONE achieved the highest average accuracy across benchmarks for both Qwen3-4B and 8B, surpassing concurrent label-free methods like u-OPSD.
- Generalization: The method proved effective across different model families, with Gemma 4 also showing gains that matched or exceeded supervised baselines.
Limitations and Future Work
- Audit Flaws: An audit by Claude 5 Opus revealed that about 20% of the "kept" stepping stones were still ill-posed or incorrectly solved. While the method is robust to some noise, improving the "stone" filtering could yield even better results.
- Task Domain: The study focused exclusively on mathematics with checkable final answers. It remains to be seen how "stepping stones" would be defined or verified for open-ended tasks like creative writing or coding.
- Computational Cost: Generating and verifying multiple attempts for stepping stones adds significant preprocessing time before the actual distillation training begins.
- Scalability: The authors evaluated models up to 8B parameters; whether these gains persist or diminish for frontier-scale models (70B+) is an open question.
Summary updated