When a gene is knocked out, an organism often shows no defect, because a related gene (a paralog, a copy left over from an ancient duplication) quietly takes over the work. LLM agents that call external tools have no such backup. If a calculator, search API, or object detector starts returning plausible but wrong answers, the agent usually accepts them. If the agent is prompted to be suspicious, it starts rejecting correct answers instead. This note proposes giving agents an explicit, measured map of which tools can stand in for which. The map is built the way geneticists find redundant genes: knock out combinations of tools and record what still works. At run time the agent uses the map to cross-check a suspect tool against its true backup only when needed, and to reroute to that backup when the primary disagrees.
The point is not "add redundancy." Running several tools and voting is old practice. The point is that knowing which tools are actually redundant is the hard part, and the obvious ways of finding out give the wrong answer. Removing one tool at a time hides redundancy entirely, because the backup absorbs the loss. Judging similarity from tool descriptions is fooled by tools that look alike but behave differently. The claim is that a multi-knockout estimate of functional overlap yields a "compensation map." That map lets an agent catch silent tool errors more often and reject correct outputs less often than checklist prompting does, at a small fraction of the cost of always-on cross-execution.
Where this sits
Silent tool failure is well documented as a gap. Tools Fail catalogs silent errors (digit swaps, sign flips, shifted detector boxes on ALFRED) and finds that agents overtrust them. In-context checklists help on arithmetic but make agents over-skeptical in embodied settings. The authors explicitly leave "recovery planning (deciding whether to retry or switch tools)" to future work, and this proposal takes up that open problem. StableToolBench motivates the setting: it reports that 55.6% of ToolBench tools were inconsistent over time. Its simulator also lets us break APIs deliberately, and the paper tests up to 50% broken.
Existing recovery systems repair either the plan or the code, not the choice of tool. OrchDAG represents tool calls as a dependency graph and trains a model with graph-edit-distance rewards to reschedule downstream calls after timeouts. It reacts to loud failures and never asks whether a different tool could do the same job. R³DAO edits the task graph with Refine, Insert and Skip operators driven by stack traces. None of these operators substitutes a functionally overlapping tool when the primary one is silently wrong. We reuse their graph view of execution and add a substitution operator whose candidates come from measurement rather than from the LLM's guess.
Failure attribution has the same blind spot. AgenTracer and Aegis assign blame by replacing one step or agent at a time, through counterfactual replay or injected faults. Both are single-substitution methods. Who Does What? showed that single lesions systematically fail in redundant networks. Its Performance Prediction Algorithm (PPA) fits a per-element contribution vector and a nonlinear prediction function from sampled multi-element lesions. On a 10-neuron network it reached 0.9978 correlation with actual performance on 230 held-out lesion configurations. We supply the method from that paper, moved from neurons to tools. The biology tells us what to expect from the measurement. The soybean paralogs LEAFY1 and 2 split labor asymmetrically: one copy does most of the work, and only the double knockout shows a phenotype. KDM5B knockout is masked by its relatives being switched on. Both point to backups that are cheap to keep and costly to run constantly. That is exactly the regime in which selective cross-checking should beat both voting and blind trust.
Core Claim
For a tool library and task family, let $P(S)$ be task success when the tool subset $S$ is disabled. Single-tool ablation sees only $P(\{i\})$, and that quantity is near baseline for any tool with a backup. Fitting the PPA form
$$\hat P(S) = f\Big(\sum_{i \notin S} c_i\Big)$$
to a sample of random multi-tool knockouts gives contributions $c_i$ (how much tool $i$ supports success) and a monotone function $f$. Pairwise redundancy is then read off as the gap between the predicted single and double knockouts. Tools $i$ and $j$ are paralogs for a task family when removing either alone costs little but removing both costs a lot. The resulting compensation map drives two run-time behaviors. The first is a selective consistency check: a primary tool's output is re-queried on its mapped paralog only when the call is high-contribution, or when a cheap disagreement probe fires. The second is rerouting: on disagreement, the agent switches to the paralog and marks the primary as degraded for the rest of the episode.
The claim is falsifiable in three places. The multi-lesion map must predict held-out double knockouts better than single-lesion or description-based maps. The map-driven agent must recover more silently failed tasks than checklist prompting while rejecting fewer correct outputs on clean runs. It must also match always-on voting at a fraction of the calls.
Research Program
The first component is lesion mapping. Inside the StableToolBench simulator, for each task category, we sample random subsets of relevant tools to disable, both by removal and by silent corruption. We run a fixed agent on the category's solvable tasks and fit $c$ and $f$. Exhaustive knockout scales as $2^N$, and PPA was designed to avoid it by learning from sampled configurations. We will report how many sampled configurations are needed before held-out prediction stabilizes, since this is the method's main cost.
The second component builds the compensation map and a substitution operator. Each tool gets a ranked list of paralogs with an estimated asymmetry, distinguishing a primary from a backup. We also separate tools that already contribute on clean runs from tools that matter only once another fails. This is the "active compensation versus passive overlap" distinction the biology literature raises, and here it is directly measurable.
The third component is the run-time policy. A Qwen2.5-7B agent, the same base model as OrchDAG's reported result, runs with an added Substitute action alongside normal calls. Cross-checks are issued according to the map and a call budget. Output agreement is judged by a type-aware comparator, numeric tolerance for numeric outputs and an LLM judge for free text. This avoids the over-skepticism that comes from asking the agent itself to doubt everything.
Baselines
The naive reference is the same agent with no recovery mechanism, run on identical corrupted and clean task sets. It measures how much silent corruption costs in the first place.
The prompting baseline is the Checklist intervention from Tools Fail, the most effective detection heuristic reported there. It is paired with an instruction to switch tools when an error is suspected, so that it can recover as well as detect. It gets the same model and the same tool access. Beating it means higher recovery on corrupted runs and a lower false-rejection rate on clean runs; winning only on the first would just reproduce its over-skepticism.
The strongest trained competitor is OrchDAG-style RL training. OrchDAG reports a 47.1% solvable pass rate on StableToolBench for Qwen2.5-7B, above GPT-4's 45.5%. That number is on clean StableToolBench. We will reproduce it, re-evaluate under our silent-corruption protocol, and give it the Substitute action with candidates chosen by the model itself. Its tuning budget and training data will match ours. Our method beats it if it recovers more corrupted tasks without losing clean pass rate.
The upper-cost reference is always-on N-version execution. Every call goes to all plausible paralogs, and the outputs are combined by majority vote. We plot success against tool calls for this baseline and for ours. The proposal wins if it reaches a comparable success rate at substantially fewer calls. If voting is both cheap enough and better, our selective policy has no reason to exist.
Two ablations isolate the key ingredient by keeping the run-time policy fixed and swapping the map. One map is built from single-tool ablations. The other comes from asking the LLM which tools are interchangeable based on their documentation. A random-paralog map serves as a floor.
Why not just…?
Why not just feed the error back and let the model retry or self-refine? Silent failures produce no error message to feed back, which is the defining property in Tools Fail. Even for loud failures, AgentRefine shows that untrained open models repeat failed actions. Retrying the same broken tool also cannot fix a tool that is consistently wrong. The no-recovery and checklist-plus-switch baselines measure how much self-correction recovers here.
Why not just hard-code fallback lists (if A fails, call B)? try/except never fires on a plausible wrong answer. A hand-written list also encodes a human guess about overlap that drifts as tools rot, given StableToolBench's reported 55.6% inconsistency rate. The description-based map ablation is effectively an automated fallback list. If it matches the lesion map, we will say that measurement is unnecessary and narrow the claim to the run-time policy.
Why not just ask the LLM which tools are similar? This is the most serious threat, because it is cheap. Evidence against it comes from the API Replacement condition in IncompleteToolBench (paper), which is built from tools that look semantically similar but are functionally wrong. Conversely, true paralogs may share no vocabulary. The head-to-head map ablation, scored by held-out double-knockout prediction and by downstream recovery, settles this directly.
Why not just vote across all tools every time? It probably works, and it is expensive in calls, latency and rate-limited APIs. The success-versus-calls curve answers whether selective checking is worth it. We expect voting to stay a useful upper bound rather than to be beaten on raw success.
Proposed Experiments
The main benchmark is StableToolBench's 765 solvable tasks, scored by Solvable Pass Rate. Silent corruptions follow the Tools Fail taxonomy of digit replacement, magnitude shifts and sign inversion for numeric outputs, plus field perturbation for structured outputs. They are applied at controlled rates through the simulator. IncompleteToolBench's API-replacement split tests whether the map avoids look-alike but wrong substitutes. The Broken Calculator set of 300 equations from Tools Fail provides a small, fully controlled check of the comparator. The original paper reports no exact pass percentages, so that setting has no reported number to compare against. The same holds for silent-corruption recovery on StableToolBench, where we found no reported result on Lacuna. All corrupted-condition numbers will therefore be new, with baselines run by us under the same protocol.
The first decisive check runs before any agent result: on held-out double and triple knockouts, does the multi-lesion map predict success better, by correlation and error, than single-lesion and description-based maps? If it does not, the rest of the program loses its premise. The second check measures recovery rate on corrupted runs together with false-rejection rate on clean runs, against Checklist. The third places the success-versus-calls curve against always-on voting. A final diagnostic plots recovery against estimated asymmetry. We expect the largest gains where a tool is a high-contribution primary with a low-contribution backup, the LEAFY-like case. Gains should be near zero where tools are symmetric and passive voting already suffices.
References
- Tools Fail: Detecting Silent Errors in Faulty Tools
- StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models
- OrchDAG: Complex Tool Orchestration in Multi-Turn Interactions with Plan DAGs
- $R^3$DAO: Reactive Recovery and Reconstruction for Long-horizon Data Agent Orchestration
- AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?
- Aegis: Automated Error Generation and Attribution for Multi-Agent Systems
- Who Does What? A Novel Algorithm to Determine Function Localization
- LEAFY1 and 2 are required for floral organ development in soybean
- The Lysine Demethylase KDM5B Regulates Islet Function and Glucose Homeostasis
- AgentRefine: Enhancing Agent Generalization through Refinement Tuning
- Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?