TL;DR
When fine-tuning LLMs on self-generated data, providing "worked solutions" as a crutch during data generation makes training cheaper but significantly impairs the model's ability to compose and reuse rules. Even when standard accuracy on practiced tasks looks identical, models trained with worked solutions struggle with sequential reasoning and become more expensive to run at inference time.
What Problem Does This Paper Address?
To avoid the latency and context-window costs of prompting with long rulebooks, researchers often "distill" rules into a model's weights. A common recipe is to have the model answer practice questions while the rules are in its context, then fine-tune the model on its own correct answers (with the rules removed).
The big question is: What should be in that "capture context" while the model writes its own training data?
Intuitively, providing a "worked solution" alongside the rules should help the model generate high-quality training data more efficiently. However, this paper investigates whether this "easy" training data hides a hidden cost: a failure to truly master the underlying logic required for complex, multi-step reasoning (composition).
Methods
The researcher studied a 2B-parameter reasoning model (MiniCPM5-2B) learning two invented quantitative rules for a fictional "Kestrel" battery cell. The experiment compared two capture policies:
- GEN (Sheet-only): The model generates training targets seeing only the rule sheet.
- INST (Sheet + Worked Solution): The model sees the rules plus a program-written worked solution.
Crucially, the model is fine-tuned only on the outputs (the reasoning traces and final answers), with the rules and worked solutions removed from the input. The model was then tested on three types of operations:
- Forward use: Applying a rule once (the practiced task).
- Inversion: Recovering a parameter from an observed value (unpracticed).
- Composition: Sequentially reusing a rule across two segments (unpracticed).
Datasets
The study uses a custom-built synthetic domain called the Kestrel cell.
- Training Set: 128 "forward" questions where the model must calculate voltage or charge for a single segment of time.
- Evaluation Sets:
- Sealed/Fresh sets: Questions using the same numerical arguments seen during training.
- New-argument set: Questions using entirely new numerical values to test for general symbolic mastery rather than memorization.
Key Insights and Ideas
- The "Crutch" Effect: Providing a worked solution acts as a cognitive crutch. The model learns to reach the correct answer by following the provided template rather than internalizing the "general route" of the rule.
- Underspecification of Accuracy: Standard accuracy on practiced tasks is a misleading metric. Two models can achieve identical "forward" accuracy while having vastly different levels of latent mastery.
- Inference Cost Inversion: While the "worked solution" policy (INST) is cheaper to train (shorter targets, less compute), it is far more expensive at inference time. Because the model hasn't mastered the rules, it generates longer, more rambling responses and fails more often on complex tasks.
- Configuration-Bound Learning: The paper suggests that models trained with worked solutions become "configuration-bound"—they can perform the specific procedure they practiced but fail when the task requires flexible reuse of that knowledge.
Key Results
The results reveal a massive "composition gap" that standard accuracy metrics completely miss:
- Identical Forward Accuracy: On unseen argument values, both policies reached 87.5% accuracy on the practiced task.
- The Composition Gap: On the unpracticed composition task, the sheet-only (GEN) model achieved 58.3% success, while the worked-solution (INST) model plummeted to 23.6%—a 34.7-point gap.
- Efficiency Trade-off: The INST policy used 42% fewer tokens during data capture and 28% less update time. However, at deployment, it required 2.6 times as many generated tokens per successful composition compared to the GEN model.
- Dose Doesn't Fix It: Even when the INST model was trained for more passes to match the total number of training tokens seen by the GEN model, a significant performance gap remained.
Limitations and Future Work
- Scope: The study was conducted in a single invented domain with one specific 2B-parameter model. It is unclear if these findings scale to much larger models (e.g., 70B+) or more diverse domains.
- The "Mediator" Problem: The worked solution changes many things at once: target length, content, and the presence of references to the solution. The paper does not isolate exactly which of these factors causes the degradation in composition.
- Future Directions: The author suggests "edit tests" to remove specific parts of the worked solution (like references) to find the exact cause of the gap. Additionally, testing on models with stronger "zero-shot" transfer capabilities could provide more power to the analysis.
Summary updated