A Cost That Practice Accuracy Hides: Worked Solutions in Self-Generated Training Data and the Compositional Use of New Rules

Jiantong XuarXiv 2026 · arXiv

TL;DR

When fine-tuning LLMs on self-generated data, providing "worked solutions" as a crutch during data generation makes training cheaper but significantly impairs the model's ability to compose and reuse rules. Even when standard accuracy on practiced tasks looks identical, models trained with worked solutions struggle with sequential reasoning and become more expensive to run at inference time.

What Problem Does This Paper Address?

To avoid the latency and context-window costs of prompting with long rulebooks, researchers often "distill" rules into a model's weights. A common recipe is to have the model answer practice questions while the rules are in its context, then fine-tune the model on its own correct answers (with the rules removed).

The big question is: What should be in that "capture context" while the model writes its own training data?

Intuitively, providing a "worked solution" alongside the rules should help the model generate high-quality training data more efficiently. However, this paper investigates whether this "easy" training data hides a hidden cost: a failure to truly master the underlying logic required for complex, multi-step reasoning (composition).

Methods

The researcher studied a 2B-parameter reasoning model (MiniCPM5-2B) learning two invented quantitative rules for a fictional "Kestrel" battery cell. The experiment compared two capture policies:

  1. GEN (Sheet-only): The model generates training targets seeing only the rule sheet.
  2. INST (Sheet + Worked Solution): The model sees the rules plus a program-written worked solution.

Crucially, the model is fine-tuned only on the outputs (the reasoning traces and final answers), with the rules and worked solutions removed from the input. The model was then tested on three types of operations:

  • Forward use: Applying a rule once (the practiced task).
  • Inversion: Recovering a parameter from an observed value (unpracticed).
  • Composition: Sequentially reusing a rule across two segments (unpracticed).
The experimental pipeline showing how capture context leads to different training targets and subsequent testing.

Datasets

The study uses a custom-built synthetic domain called the Kestrel cell.

  • Training Set: 128 "forward" questions where the model must calculate voltage or charge for a single segment of time.
  • Evaluation Sets:
    • Sealed/Fresh sets: Questions using the same numerical arguments seen during training.
  • New-argument set: Questions using entirely new numerical values to test for general symbolic mastery rather than memorization.

Key Insights and Ideas

  • The "Crutch" Effect: Providing a worked solution acts as a cognitive crutch. The model learns to reach the correct answer by following the provided template rather than internalizing the "general route" of the rule.
  • Underspecification of Accuracy: Standard accuracy on practiced tasks is a misleading metric. Two models can achieve identical "forward" accuracy while having vastly different levels of latent mastery.
  • Inference Cost Inversion: While the "worked solution" policy (INST) is cheaper to train (shorter targets, less compute), it is far more expensive at inference time. Because the model hasn't mastered the rules, it generates longer, more rambling responses and fails more often on complex tasks.
  • Configuration-Bound Learning: The paper suggests that models trained with worked solutions become "configuration-bound"—they can perform the specific procedure they practiced but fail when the task requires flexible reuse of that knowledge.

Key Results

The results reveal a massive "composition gap" that standard accuracy metrics completely miss:

  • Identical Forward Accuracy: On unseen argument values, both policies reached 87.5% accuracy on the practiced task.
  • The Composition Gap: On the unpracticed composition task, the sheet-only (GEN) model achieved 58.3% success, while the worked-solution (INST) model plummeted to 23.6%—a 34.7-point gap.
  • Efficiency Trade-off: The INST policy used 42% fewer tokens during data capture and 28% less update time. However, at deployment, it required 2.6 times as many generated tokens per successful composition compared to the GEN model.
  • Dose Doesn't Fix It: Even when the INST model was trained for more passes to match the total number of training tokens seen by the GEN model, a significant performance gap remained.
Comparison of success rates between the two policies, highlighting that while forward use is similar, composition reveals a large performance gap.

Limitations and Future Work

  • Scope: The study was conducted in a single invented domain with one specific 2B-parameter model. It is unclear if these findings scale to much larger models (e.g., 70B+) or more diverse domains.
  • The "Mediator" Problem: The worked solution changes many things at once: target length, content, and the presence of references to the solution. The paper does not isolate exactly which of these factors causes the degradation in composition.
  • Future Directions: The author suggests "edit tests" to remove specific parts of the worked solution (like references) to find the exact cause of the gap. Additionally, testing on models with stronger "zero-shot" transfer capabilities could provide more power to the analysis.

Summary updated

Abstract

A worked solution can make self-generated training targets shorter and cheaper, but accuracy on the practiced operation may hide how well a model can reuse newly learned rules. We study this choice in a 2B-parameter reasoning model learning two invented quantitative rules. The model generates targets from a rule sheet alone or from the sheet plus a program-written worked solution; targets are accepted for final-answer correctness, and the support is removed for fine-tuning and evaluation. Training questions request forward use in one segment; composition requires sequential reuse of the same rule across two segments. After six passes, using a program-checked numerical signature, sheet-only targets lead by 24.3 percentage points (95% item-bootstrap CI 14.6-34.0) on composition in a prospective test. On unseen argument values, observed forward accuracy is identical (87.5% each), but composition accuracy is 58.3% versus 23.6%, a 34.7-point gap (25.7-43.8). An independently sampled second target-bank pair repeats the direction (+14.6 points, 2.1-26.0). Ten passes on the shorter worked-solution targets, approximately matching the domain tokens of six sheet-only passes while changing other training quantities, narrow the gap to 16.3 points (6.3-26.7) in the first bank pair and 6.2 points (-6.3 to 18.8) in the second, where the pre-specified test does not hold. At six passes each, the worked-solution policy uses fewer capture tokens and less update time, yet the trained models use 2.6 times as many generated tokens per composition success under the numerical signature (2.1 times in the second bank pair). Choosing a capture context therefore requires testing how the new rules are reused, alongside the inference cost of those uses, rather than relying on forward accuracy alone.
BibTeX
@misc{xu2026cost,
  author = {Jiantong Xu},
  title = {{A Cost That Practice Accuracy Hides: Worked Solutions in Self-Generated Training Data and the Compositional Use of New Rules}},
  year = {2026},
  eprint = {2611.12225},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12225},
}
Figure