Referee Disagreement as the Escalation Signal for Vetting Composed Tool Plans

Language-model agents now build their own multi-step programs out of thousands of web APIs. They pick services, chain their outputs, and call them live. Before execution, nothing checks whether a plan is missing a required input, relies on a look-alike tool that cannot do the job, or pulls in a tool package that is malicious. Checking every plan with a strong model is accurate but expensive. Checking every plan with a cheap scanner is fast but breaks on unfamiliar inputs. Academic editorial boards have long handled the same trade-off with a specific rule. Several independent referees read each submission. The editor decides alone when the referees agree, and a more senior editor steps in only when they disagree.

This note proposes transplanting that rule into automated service composition, meaning the automatic assembly of a workflow from independent services. Several cheap, deliberately different checkers vet each step of a composed plan. When they agree, their verdict stands. When they disagree, the plan goes up to one expensive specialist model. The bet is that, on inputs unlike anything seen in training, disagreement between different kinds of checkers flags errors more reliably than any single checker's own confidence. If that holds, the system gets close to strong-verifier accuracy while calling the strong verifier on only a minority of plans.

Core Claim

The claim is narrow and falsifiable. Consider pre-execution vetting of tool plans, which has two parts: deciding whether to execute or abstain when a condition is incomplete, and deciding whether to admit or reject a tool package. For this task, escalation triggered by disagreement among heterogeneous checkers is compared with escalation triggered when a single cheap model is unsure, at the same escalation rate. We predict that disagreement-triggered escalation gives higher accuracy, and a much lower false-alarm rate on benign inputs from unseen sources.

Two pieces of evidence in the literature motivate this. The first is that cheap checkers are confidently wrong out of distribution. On MaliciousSkillBench, the best learned detector keeps 95.6% malicious recall on held-out sources, but it flags 62.4% of benign skills. Off-the-shelf scanners do the opposite: they catch at most 2.5% of malicious skills from unseen sources. These two tools fail in opposite directions. That is exactly the situation in which their disagreement carries information and neither one's confidence does.

The second is that agents are over-eager to act. Can Tool-augmented Large Language Models be Aware of Incomplete Conditions? shows that models call tools even when a parameter is missing or the right tool is absent. It also shows that a structured five-step checklist lifts a 4B model from 20.90% to 68.57% F1 on the ToolBench API-replacement split. A checklist-prompted small model is therefore a credible cheap referee rather than a straw man.

Relation to Prior Work

Hierarchy already appears in tool composition, but only as a way to narrow search. AnyTool sorts 16,000+ APIs into category, tool and endpoint agents, then routes solver failures back up to those agents to retry retrieval. Its upper tiers never approve or veto a plan before it runs. We keep AnyTool as the plan generator and add a vetting layer that it lacks. ToolLLM contributes the benchmark and the tree-search solver, which backtracks after errors, but it too executes without review.

Cooperative Service Composition lets provider agents bid on requests using self-reported capability descriptions and quality figures. Principal-agent analyses of vertical delegation, such as the apprenticeship governance study and the bank credit-risk study, warn that self-reports from delegated parties are exactly where misrepresentation enters. The bank study further argues that oversight based on formalities fails without objective checks. Together these constrain our design: the checkers must not all read the same self-described text, or they will be fooled together. That is why one checker is purely structural.

The organizational literature supplies the architecture. The tiered referee, associate-editor and editor-in-chief roles in IEEE editorial structures (IEEE Transactions on Cybernetics) set the template. ARGO describes a governance layer made of shared baselines plus shared tooling. It is a qualitative case study with no executable enforcement, and the regulatory compliance survey names automated operational compliance as underexplored. We treat the editor-in-chief tier as a written policy compiled into checks, not as another model.

Embodied LLM Agents Learn to Cooperate in Organized Teams finds that a designated leader improves agent teams, cutting task completion time by 9.76% for three GPT-3.5 agents. That result motivates structure, but it concerns household coordination, not verification. Finally, Dirigent shows that every added coordination layer costs latency in high-throughput orchestration. This is why escalation has to be rare, not universal.

Research Program

We will build a three-tier vetting layer that sits between a plan generator and live execution.

The referee tier has three checkers chosen to fail differently.

  1. A structural checker. It is deterministic. It reads each step's declared inputs, outputs and required parameters from the API schema, in the spirit of the inputs-outputs-preconditions-effects profile of DAML-S. It then checks two things: whether every required parameter is bound from the user's request or an earlier step's output, and whether the chosen endpoint's declared function matches what the step asks of it.
  2. A small checklist-prompted model. This follows the structured verification recipe from the incomplete-conditions paper.
  3. A lexical content checker. This is a word-level classifier over tool descriptions and packages, trained as in MaliciousSkillBench.

Each referee returns accept or reject for each step. Unanimous verdicts are final.

The associate-editor tier is a single strong model. It sees the plan, the referees' split verdicts and their stated reasons, and it decides. The editor-in-chief tier is a short policy file listing forbidden action types and required abstentions, compiled into rules that referees and the associate editor must apply.

We will analyze three things.

  1. Disagreement versus confidence. On unseen sources, does referee disagreement flag errors more reliably than a single referee's confidence?
  2. Heterogeneity. How much of the benefit comes from using different kinds of referees?
  3. Cost against error. How does the escalation rate trade off against error rate, measured in strong-model calls and added latency per plan?

Baselines

Do nothing. The first reference point is the undefended pipeline: AnyTool or DFSDT plans executed with no vetting. It sets the floor for invalid live calls and the ceiling for pass rate. As reported numbers for this setting, AnyTool reaches a 58.2% average pass rate on filtered ToolBench and 73.8% on AnyToolBench (AnyTool). Vetting succeeds end to end if it keeps pass rate within two points of these figures while cutting invalid invocations.

Always escalate. The strongest competitor on accuracy sends every plan to the strong model with the structured-verification checklist. This is the natural upper bound, because it is what our associate editor does but applied to everything. The incomplete-conditions paper reports GPT-4o and Qwen2.5-72B as top performers. Their exact scores appear only in a figure, so no reported number is found on Lacuna for them. The reference points available are a human accuracy of about 81% and the 68.57% F1 of checklist Gemma-3-4B on ToolBench API replacement. We will rerun this baseline ourselves on identical splits with the same prompt. Beating it means matching its F1 within two points while escalating at most 30% of plans.

Confidence cascade. This ablation isolates the key ingredient. The checklist small model decides alone and escalates when its own confidence is below a threshold. The threshold is tuned so that it escalates exactly as often as our system does. It gets the same strong model and the same development-set tuning budget.

Homogeneous panel. A second ablation uses three samples of the same small model instead of three different referee types. It tests whether heterogeneity, not mere voting, carries the effect.

Static scanners alone. Cisco and SkillSpector act as the gate by themselves. They are reported to detect at most 2.5% of malicious skills on source-disjoint data.

Why not just…?

Why not just run the strong verifier on every plan? This is the real threat on accuracy, and we expect it to win or tie on F1. The proposal only matters if strong-model calls are a meaningful share of cost and latency. Dirigent argues this for high-throughput orchestration. For a single interactive agent, however, one extra call may be negligible next to a multi-call search. We will report cost per plan relative to the solver's own cost. If always-escalate adds under about 10% overhead, we will say the cascade is not worth it in that regime.

Why not just escalate on the small model's uncertainty? That is a standard cascade, and if it matches ours at equal escalation rate, our contribution reduces to known practice. The MaliciousSkillBench result gives the reason to expect it to fail out of distribution: a detector with 62.4% benign false-alarm rate is flagging benign skills confidently, not uncertainly. The confidence-cascade ablation on source-disjoint splits settles this directly.

Why not just fine-tune the tool agent to abstain? API-Bank shows that cheap synthetic data can lift a 7B model from 15.19% to 39.58% overall correctness. Abstention training could close part of the incomplete-conditions gap. It would not address malicious tool packages from new sources, and it bakes the policy into weights instead of an editable rule file. We will include a fine-tuned abstaining solver as an additional baseline on the incomplete-conditions splits only.

Proposed Experiments

Experiment 1: execute-or-abstain vetting. This runs on all four ICT splits (API replacement and utterance removal, for both API-Bank and ToolBench), a total of 1,610 instances. Metrics are F1, escalation rate and strong-model calls. To test the out-of-distribution claim, we tune thresholds on the API-Bank splits and test on the ToolBench splits.

Experiment 2: tool-package vetting. This uses MaliciousSkillBench under both random and source-disjoint protocols, with macro-F1, malicious recall and benign false-alarm rate as metrics. The decisive check here is whether disagreement-triggered escalation brings the benign false-alarm rate on held-out sources far below the detector's 62.4% while keeping recall near 95.6%, at the same escalation rate as the confidence cascade.

Experiment 3: end to end. We gate AnyTool's plans on filtered ToolBench and AnyToolBench and measure pass rate, invalid live calls and added latency. To create realistic incomplete conditions in live runs, we inject perturbations in the style of the ICT splits.

Decisive result and failure criteria. The decisive result is the comparison between disagreement-triggered escalation and the confidence cascade on source-disjoint data at matched escalation rates. The idea is refuted if the confidence cascade matches it within noise. It is also refuted if the homogeneous panel matches the heterogeneous one, which would mean voting, not diversity, did the work.