When a general language model is fine-tuned hard on one narrow domain, it can lose skills it used to have. The most striking recent case is a math theorem prover whose ability to call general software tools fell from 89.4% to about 4.6% after training on formal proofs. A hundred tool-using example traces brought most of that ability back. The repaired model then called tools even when no tool was relevant. We propose that this "trigger-happy" side effect is predictable. Tool use is not one skill. It is a stack of three decisions: whether a tool should be called at all, which step of the plan needs it, and how to fill in the call. Repair data made only of successful calls trains the bottom of that stack and overwrites the top.
The proposal has three parts. First, an automatic judge locates each failure of the specialized model at the lowest of these three levels where it breaks. Second, the repair data is built from short, minimal corrections aimed at those levels, including "don't call anything" corrections. Third, we test whether this repairs tool use without trading away abstention or domain skill. The four directions play different roles here. Forgetting is the phenomenon. A three-level task hierarchy is the unit of diagnosis and of data allocation. The automatic judge is the measuring instrument, and its noise is a constraint. The finding that dense feedback hurts learners motivates keeping each correction minimal and localized.
Core Claim
Awakening the Sleeping Agent motivates the claim. It showed that tool-use collapse in Goedel-Prover-V2 is suppression rather than erasure. It restored general function calling on the Berkeley Function Calling Leaderboard (BFCL v4) to 83.8% and raised ProofNet pass@32 from 21.51% to 25.81%. It also reported a sharp drop in irrelevance detection, the benchmark category where the correct answer is to call no tool, with 99.1–99.9% of generations attempting a call.
Our claim is that forgetting and reactivation happen unevenly across levels of the task hierarchy. A flat success metric hides this. We use the three levels of ToolBeHonest, a benchmark that separates deciding whether a goal is solvable (Level 1), finding the blocking sub-goal (Level 2), and specifying what the missing tool must do (Level 3). Repair data allocated by a per-level diagnosis of the specific specialized model should recover Level-3 call execution as well as uniform trace fine-tuning does, without the Level-1 loss.
What would be new is using a hierarchical taxonomy as a training-time allocation rule for repairing forgotten abilities. Existing hierarchical benchmarks such as ToolBeHonest and BFCL are used only as after-the-fact scorecards. Existing forgetting remedies, the pruning regularizer RECT and the neuron-masking editor NMKE, operate on flat factual edit streams (ZsRE, CounterFact) with clean targets.
Research Program
The first step is a per-level forgetting profile. We run the base general model, the collapsed specialist, and a reproduced version of the Chung et al. reactivated specialist through BFCL v4, including its irrelevance category. We also run ToolBeHonest at all three levels and under all three incomplete-toolset conditions (missing necessary tools, potential tools, and limited-functionality tools). Finally we run the incomplete-conditions benchmark ICT from Can Tool-augmented LLMs be Aware of Incomplete Conditions?, which has 1,610 instances across 57 domains built from APIBank and ToolBench.
We report retention separately per level, measured as the specialist's score divided by the base model's score. Every checkpoint is then described by a three-number profile instead of one number. This step alone tests whether the trigger-happy effect is a Level-1 loss hiding under Level-3 recovery.
The second step is failure localization. For each failed item from the specialist, we assign the lowest failing level, and we use deterministic checks wherever they exist. Level 1 is mostly mechanical: the model either did or did not emit a call when the gold answer abstains. Level 3 can use BFCL's syntax-tree and execution matching. An LLM judge is needed only to decide whether a plan identified the right blocking sub-goal (Level 2), in the style of the two-stage decomposed judging in VALOR-EVAL.
Judge noise is a real constraint. Curriculum-RLAIF found that AI-judge accuracy drops steeply on items humans find ambiguous. Evaluating Open-QA Evaluation found that weaker judges score vague answers as correct. We therefore validate the Level-2 judge against human labels on a held-out sample of a few hundred items and report agreement. Items where the judge's confidence is low are excluded from allocation rather than guessed.
The third step builds the repair data. Following the general-scaffolder-plus-specialist trace synthesis of Chung et al., we generate candidate traces in the Lean domain. We then allocate a fixed budget of 100 traces in proportion to the specialist's diagnosed per-level deficit. Level-1 slots hold in-domain traces where the correct action is to stop and state that the goal is not solvable with the available tools. Level-2 slots hold traces that name the blocking step. Level-3 slots hold correct calls.
Each corrective trace is kept structurally minimal. It contains only the decision at the failing level, not a full execution log. This follows PGS, which shows that stripping feedback down to property checks and minimal counterexamples fixes more bugs than verbose traces. It also follows the "instruction paradox" in United Minds or Isolated Agents?, where detailed guidance helps hard tasks but hurts easy ones. Both findings come from inference time; whether minimality matters during training is one of the things we test.
As a second setting, we repeat the pipeline on a small general model fine-tuned to collapse on a narrow tool domain drawn from ICT. This checks that the effect is not specific to one prover.
Baselines
The naive reference is the collapsed specialist, which scores about 4.6% on BFCL. Next to it we report the unspecialized base model, 89.4% per Chung et al., as the ceiling.
The strongest existing method is the uniform reactivation of Awakening the Sleeping Agent. It fine-tunes on 100 synthesized Lean agentic traces and reaches 83.8% on BFCL v4 and 25.81% ProofNet pass@32. It is the strongest competitor because it already recovers most tool use with tiny data while improving the domain. The file reports that irrelevance detection drops but gives no figure: no reported result found on Lacuna for the reactivated model's irrelevance-category score. We will therefore reproduce it ourselves.
We run it with the same trace generator, the same 100-trace budget, the same optimizer, and the same number of tuning trials as our method, on the same checkpoint. Beating it means two things together. First, BFCL overall within two points of its reproduced score and ProofNet pass@32 no lower. Second, a statistically significant gain on BFCL irrelevance and on ToolBeHonest Level 1, with bootstrap intervals over three seeds.
The strongest inference-only competitor is Structured Verification from the ICT paper, a five-step checklist prompt run before any call. The paper reports it raising Gemma-3-4B's F1 on ToolBench API-Replacement from 20.90% to 68.57%, against a human reference of about 81%. We apply it both to the uniformly reactivated model and to ours. On ToolBeHonest, GPT-4o's reported overall score of 37.0 and Gemini-1.5-Pro's 45.3 give context for how hard Levels 1–3 are.
Three ablations isolate the key ingredient. The first uses the same number of abstention and planning traces, but drawn uniformly instead of by diagnosis. This separates "having negative examples" from "diagnosis-driven allocation." The second uses per-level allocation with verbose rather than minimal traces, which isolates feedback density. The third uses judge labels replaced by random level labels at matched marginals, which tests whether judge quality matters.
We also combine our data with RECT-style pruning of the fine-tuning update, keeping only the largest relative weight changes as in RECT. This shows whether the two approaches stack. General-ability drift is tracked on GSM8K, HumanEval, and MMLU throughout.
Why not just…?
Why not just add some "no tool needed" examples to the 100 traces? This is the most serious threat, and it may be enough on its own. Our first ablation is exactly this test. If uniformly sampled abstention examples at matched count give the same per-level profile as diagnosis-driven allocation, the taxonomy adds nothing beyond the data mix. We would then report the simpler recipe and treat the per-level forgetting profile as the main contribution. The case for diagnosis rests on the expectation that different specialists lose different levels. The second setting tests this directly: if the two models' profiles look the same, uniform mixing wins.
Why not just prompt the repaired model with Structured Verification? Prompting cannot help the collapsed specialist, which barely emits calls at all. On the repaired model, a checklist might restore abstention without any retraining. The instruction-paradox result in United Minds or Isolated Agents?, however, predicts that a fixed multi-step checklist hurts simple, well-formed queries. We measure this directly: accuracy and output length on BFCL's simple single-call category, with and without the checklist, for both repaired models. If the checklist restores Level 1 at no cost on simple calls, that is a cheaper answer and we will say so.
Why not just merge the specialist with the base model? Chung et al. tested spherical weight interpolation between the collapsed prover and its general base. It brought back some tool use but caused a large drop in math, which they describe as zero-sum. We include one interpolation sweep at matched evaluation so the tradeoff curve is visible. We do not expect it to compete.
Decisive Experiments
The first decisive check is the per-level profile of the reproduced uniform reactivation. If Level 1 of ToolBeHonest and BFCL irrelevance are not clearly below the base model while Level 3 is near it, the premise fails and the project stops there.
The second is the head-to-head at a fixed budget of 100 traces. It compares our allocation, uniform reactivation, and the matched uniform-negatives ablation on BFCL overall and irrelevance, ToolBeHonest Levels 1–3, ICT F1, and ProofNet pass@32. The idea succeeds only if it beats uniform negatives on Level 1 without losing BFCL or ProofNet.
The third checks whether the judge can be trusted where it matters. We report human agreement with the Level-2 judge and rerun allocation with random level labels. If random labels do as well, the judge is not doing useful work.
The fourth varies trace density at fixed allocation. It tests whether the feedback-minimality result from inference-time repair also holds as a training-time effect on how much a model forgets. We measure this as drift on GSM8K and HumanEval per point of BFCL gained.
References
- Awakening the Sleeping Agent: Lean-Specific Agentic Data Reactivates General Tool Use in Goedel Prover
- ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models
- Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue
- Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs
- Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?
- VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
- Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback
- Evaluating Open-QA Evaluation
- PGS: Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
- United Minds or Isolated Agents? Exploring Coordination of LLMs under Cognitive Load Theory