TL;DR
B-OPSD (On-Policy Self-Distillation with Self-Generated Problem Briefs) is a novel training method that allows Large Language Models (LLMs) to improve their reasoning capabilities without needing ground-truth solutions. By generating its own high-level "problem briefs" to serve as privileged context for a teacher role, the model provides itself with dense, token-level guidance, outperforming standard self-distillation methods on elite math benchmarks like AIME and HMMT.
What Problem Does This Paper Address?
In the quest to make LLMs better at complex reasoning (like math or coding), researchers often use On-Policy Distillation (OPD). Usually, this involves a large "teacher" model guiding a smaller "student" model. To save on compute, On-Policy Self-Distillation (OPSD) was developed, where the same model plays both roles.
However, OPSD has a major "cheat code" requirement: the teacher role is only effective because it is shown the ground-truth solution. This creates a bottleneck. If you have a million math problems but no verified answers, you can't use standard OPSD. Furthermore, outcome-based rewards (simply telling a model if the final answer is right or wrong) are "sparse"—they don't tell the model which specific reasoning step was the mistake. B-OPSD aims to provide "dense" (token-by-token) feedback without ever seeing a correct answer during training.
Methods
The core innovation of B-OPSD is the Problem Brief. Instead of a full solution, the model generates a strategic summary of the problem from the question alone.
- The Brief Generation: Using a specific prompt, the model identifies the target, relevant mathematical principles (formulas, conditions), a suggested approach (subgoals), and potential pitfalls. Crucially, the prompt forbids the model from providing the final answer.
- Two Roles, One Model:
- The Student: Sees only the problem statement and generates a solution trajectory.
- The Teacher (Frozen): Sees the problem plus the generated brief. Because it has this "privileged context," its probability distribution for the next token is generally more accurate than the student's.
- Distribution Matching: The student is updated by trying to minimize the KL divergence between its next-token distribution and the teacher's. This provides a learning signal at every single token of the reasoning process.
- Static vs. Refreshed Briefs: The authors tested two versions. B-OPSD uses "static" briefs generated once at the start. B-OPSD* uses "refreshed" briefs, where the model periodically updates its briefs as it becomes a more capable student, creating a self-improving feedback loop.
Datasets
- Training: The models were trained on 30,000 mathematical problems sampled from the OpenThoughts reasoning subset.
- Evaluation: The researchers used three high-difficulty competition math benchmarks: AIME 2024, AIME 2025, and HMMT 2025 (Harvard-MIT Mathematics Tournament).
- Models: Experiments were conducted across three sizes of the Qwen3 family: 1.7B, 4B, and 8B parameters.
Key Insights and Ideas
- Strategy > Solutions: One of the most surprising findings is that B-OPSD often outperforms OPSD variants that do have access to ground-truth solutions. This suggests that high-level strategic guidance (principles and pitfalls) might be a more effective "teacher context" than a specific, step-by-step reference solution, which might contain idiosyncratic logic that confuses the student.
- The Power of "Dense" Feedback: By matching distributions at every token, the model learns the logic of the path, not just the destination. This is significantly more sample-efficient than Reinforcement Learning with Verifiable Rewards (RLVR), which only sees the final result.
- Self-Correction via Refreshing: The "refreshed" variant (B-OPSD*) consistently performed best. As the student gets smarter, the briefs it generates become more sophisticated, which in turn provides better guidance for the next round of distillation.
Key Results
B-OPSD achieved higher average accuracy than the base model, SFT, GRPO, and standard OPSD across all tested sizes.
- Qwen3-1.7B: The base model started at 37.1% average accuracy. B-OPSD (refreshed) boosted this to 42.9%*, notably beating standard OPSD (40.9%).
- Qwen3-8B: B-OPSD reached 66.8%* average accuracy, compared to 61.8% for the base model and 65.5% for standard OPSD.
- Ablation Success: Removing the "Relevant Principles" section from the brief caused the largest drop in performance, confirming that conceptual scaffolding is the most critical part of the teacher's privileged context.
Limitations and Future Work
- Domain Specificity: The paper focuses entirely on competition mathematics. It remains to be seen if "problem briefs" are as effective for subjective tasks like creative writing or open-ended coding projects where "principles" are harder to define.
- Brief Quality: If the model is too weak to generate a somewhat helpful brief, the teacher's guidance could become "the blind leading the blind." The authors note they did not filter or verify the generated briefs, leaving room for improvement via quality control.
- Future Directions: The authors suggest exploring how often briefs should be refreshed and testing the method on broader reasoning tasks beyond the math domain.
Summary updated