Using Retriever Disagreement to Catch Near-Miss Tool Calls

Language-model agents fail at tool use in a specific, repeatable way. The user asks for something, the retriever returns a tool whose description sounds right but whose parameters do not fit the request, and the model calls it anyway. Retrieval benchmarks count this as a ranking error. Hallucination benchmarks count it as a fabricated or unavailable tool call. We argue these are the same event: a near-miss tool, meaning one whose description matches the intent while its interface does not. We propose to detect near misses at retrieval time. Two retrieval channels that are trained to rely on different evidence are compared, and when they disagree the system declines to call a tool or asks the user a question. One channel matches words and structured signature fields. The other matches meaning.

The concrete idea fits in two sentences. A small code model first drafts the function call it expects to need. Exact-token search matches the parameter names and types in that draft against the parameter names and types in the tool registry, while a separately trained neural encoder matches the request against tool descriptions. When the two channels point to different tools, that disagreement is a cheap and calibrated warning that the top tool is a near miss. The warning is reliable only if the neural encoder was trained not to copy the exact-token channel, and that dependency is the testable scientific claim of this proposal.

Core Claim

Hybrid retrieval, meaning a system that combines sparse (exact-term, BM25-style) scores with dense (neural embedding) scores, is normally judged by ranking quality alone. On Complementarity Objectives for Hybrid Retrieval shows that dense retrievers trained alongside sparse ones tend to mimic the sparse model's easy predictions, which it calls the "Intersection Trap." Its remedy is an orthogonality loss plus masking of exact keyword matches during dense training, and the paper validates this on passage retrieval only. We make a different use of the same observation. If the two channels are made to depend on different evidence, their agreement becomes an independent second opinion about whether retrieval succeeded. Our central prediction is that agreement between the channels detects near-miss tools substantially better when the channels are trained to be complementary than when they are not, even if the two systems have similar ranking scores. If that prediction fails, the proposal reduces to "hybrid retrieval for tools," and a skeptic's simpler baseline wins.

The second claim concerns what the sparse channel should index. Tool queries share few words with tool documentation. Retrieval Models Aren't Tool-Savvy reports this as the reason strong general retrievers fail on its 43k-tool TOOLRET benchmark. Running BM25 over raw descriptions therefore adds little. Parameter keys, types and enumerated values are highly specific to each tool, and they are exactly where near-miss tools differ from correct ones. The ICT benchmark's "API Replacement" perturbation, which swaps in a semantically similar but wrong tool, is built on that difference (Can Tool-augmented LLMs be Aware of Incomplete Conditions?). A query in plain English contains no parameter keys, so we let a small model write a draft call and use the draft as the sparse query. A drafted call that matches no schema at all is a direct signal of a phantom tool, the failure API-Bank documents in fine-tuned open models. In effect, the model's tendency to hallucinate is turned into the query.

Domain-specific pre-training has a narrow, testable role here. It is meant to make the drafts use realistic parameter vocabulary. We continue pre-training a small code model on raw OpenAPI specifications, following the evidence from MonoCoder and A Comparative Study of PEFT Methods for Python Code Generation that exposure during pre-training matters more than the choice of fine-tuning method. If the pre-trained drafter does not beat an instruction-tuned drafter of the same size, we will drop it.

Research Program

The system has four parts. The first is a signature drafter: a model of roughly 1B parameters that writes a hypothesized call, including function name, parameter keys and literal values, from the user request. We train two versions for comparison. One receives continued pre-training on OpenAPI specifications from API Pack's 11,213 APIs, excluding every API that appears in any evaluation set. The other is the same base model with only instruction tuning.

The second part is a structured sparse index. For each tool, its schema is broken into separate fields: name, parameter keys, types, required flags and enumerated values. The index scores how well the draft's fields overlap with each tool's fields. Following MonoCoder's Local Semantics Elimination, which anonymizes identifiers so the model learns structure, we also build a variant that replaces concrete literal values in the draft with type placeholders. This makes the channel reward interface shape rather than surface names.

The third part is a dense encoder fine-tuned on TOOLRET-train's 200k contrastive examples. It is trained with and without the orthogonality loss and keyword masking from the complementarity paper, applied against the structured sparse channel.

The fourth part is a gate. It is deliberately a small logistic model, so that any gain can be credited to the channel features rather than to the capacity of the gate. Its inputs are each channel's top score and score margin, whether the two channels' top-1 tools agree, the rank each channel assigns to the other channel's top-1, and the schema-match coverage of the draft. From these it predicts one of three outcomes: call the fused top tool, abstain because no suitable tool exists, or ask the user because a required argument is missing.

The analysis that tests the core claim uses the Ratio of Complementarity metric from the complementarity paper. We measure how the gate's area under the ROC curve for detecting a near-miss or missing tool changes as complementarity is varied, holding ranking quality roughly fixed. We will also report how often the gate's warnings fall on the tool-use steps that AgentHallu identifies as the hardest to attribute afterwards (11.6% step-localization accuracy for the best models). The point is that a warning raised at retrieval time localizes the error by construction.

Baselines

The naive reference point is a general dense embedding model used zero-shot for retrieval, with the generator always calling the top tool. On TOOLRET, Retrieval Models Aren't Tool-Savvy reports nDCG@10 of 33.83 for NV-Embed-v1, and we reproduce this configuration first. The strongest retrieval competitor is a dense retriever fine-tuned on TOOLRET-train with the same paper's target-aware instructions. That paper reports 10–20% gains in agent pass rate, but the file gives no nDCG number for it, so no reported result was found on Lacuna for fine-tuned TOOLRET nDCG and we will measure it ourselves. The simple hybrid competitor combines BM25 over raw descriptions with this fine-tuned dense model, using the linear interpolation of Yi Luan et al. with weights tuned on a held-out split. No reported result was found on Lacuna for BM25 or for this hybrid on TOOLRET.

For abstention, the strongest competitor is Structured Verification prompting. On ICT's ToolBench API Replacement split it raises Gemma-3-4B from 20.90% to 68.57% F1, against a human reference of about 81%. We run it with the same 4B-class generator used in our pipeline. For end-to-end calling, we compare against Gorilla's retriever-aware training (Gorilla), which reports a 6.98% hallucination rate on TorchHub and 71.68% zero-shot accuracy on HuggingFace in APIBench. On API-Bank we compare against the reported 60.24% total correctness for GPT-4 and 39.58% for Lynx.

Every trained baseline receives the same TOOLRET-train data, the same encoder size, the same hyperparameter search budget and the same held-out tuning split. Every threshold is tuned on the same development data as our gate.

Three ablations isolate the key ingredients. The first removes the orthogonality loss while keeping everything else. The second replaces drafted-signature sparse queries with BM25 over raw descriptions. The third swaps the pre-trained drafter for the instruction-tuned drafter. To count as beating the baselines, the full system must exceed Structured Verification F1 on ICT API Replacement and must match or exceed the fine-tuned dense retriever's nDCG@10 on TOOLRET. The orthogonality ablation must also show a clearly lower gate area under the ROC curve at similar nDCG.

Why not just…?

Why not just fine-tune a dense retriever on TOOLRET-train and stop? This is the most serious threat to the ranking claim, and we expect it to close much of the nDCG gap. Our claim is not mainly about ranking, however. A single dense channel supplies only its own score margin as a confidence signal, and near-miss tools are by definition close in description space, so that margin should be small exactly when it matters most. We include "fine-tuned dense plus margin threshold" as a gate baseline. If it matches our gate on ICT, the second channel is not justified.

Why not just fuse BM25 and dense scores, with no special training? Linear fusion is cheap and often strong. The complementarity paper's Intersection Trap result predicts that untrained fusion yields correlated channels, which would make their agreement uninformative. The orthogonality ablation tests this directly. If untrained fusion yields a gate that is just as good, we will report that complementarity training is unnecessary for this purpose.

Why not just prompt the model to verify before calling, or use retriever-aware training? Structured Verification already recovers much of the abstention gap. Its remaining distance from human performance (68.57% against about 81%) and its per-query reasoning cost leave room for a retrieval-time signal. Gorilla's training teaches the model to ignore irrelevant documents, but it was evaluated on complete instructions and does not address missing arguments. The approaches can be combined, so we also test our gate stacked with Structured Verification. If the stack does not beat Structured Verification alone, our signal is redundant with prompting.

Why not just continue pre-training the retrieval encoder itself on API specs? This is plausible, and it is the natural reading of the gap between general and tool retrieval. We include it as an extra arm in which the dense encoder receives the same OpenAPI continued pre-training. If that arm gives the same detection gains without a drafter, the drafter-based sparse channel is unnecessary.

Proposed Experiments

Retrieval quality is measured by nDCG@10 and Recall@k on TOOLRET's 43k-tool corpus. Detection is measured by the gate's area under the ROC curve and by call/abstain/ask F1 on ICT (resource ddehun/ICT). Because ICT is derived from API-Bank and ToolBench, we remove any tool that overlaps with TOOLRET-train from both training and gate tuning. End-to-end behavior is measured by Level-2 retrieve-and-call correctness in API-Bank's executable environment, and by AST accuracy and hallucination rate on APIBench (resource ShishirPatil/gorilla), with our retriever replacing Gorilla's.

The decisive check is a sweep over the weight on the orthogonality loss. For each setting we plot the Ratio of Complementarity against the gate's area under the ROC curve on ICT API Replacement, and we separately report nDCG@10. The idea holds if detection rises with complementarity while nDCG stays roughly flat, and if the full system clears Structured Verification's 68.57% F1 with the same generator. It fails if detection tracks nDCG and not complementarity. In that case we would recommend the simpler fine-tuned dense retriever, and the proposal would stand as a careful negative result. A second check compares drafted-signature sparse queries with raw-description BM25 on the subset of ICT where the substituted tool shares its description but differs in its parameters. This is where the structured channel should matter most.