Harm-Mined Contrastive Retrieval of Tool Documentation for Agents

Tool-using language model agents need API documentation in their context, but more documentation is not better. On enterprise API tasks, giving agents the relevant documentation raises pass rates by 7.0 percentage points, yet giving them all of it dilutes the context and hurts multi-step tasks (Agent-Diff). Single-turn studies show the same pattern in sharper form. A few coherent but irrelevant sentences, chosen to push a model toward a specific wrong answer, cut frontier model accuracy by over 45%, while generic noise barely registers (Adaptive Distraction). This proposal trains a small retriever that picks which documentation pages an agent sees at each step. The retriever is taught what to exclude using documents that measurably break the agent's execution, not documents that merely look similar to the query.

The approach connects three lines of work. Interactive benchmarks judge success by checking the environment's database before and after the agent acts (a "state diff"). That check gives an exact, cheap signal for whether showing a document made the agent fail. Contrastive training, which teaches an encoder to score the right document above wrong ones, usually needs very large batches so that each batch contains wrong documents worth learning from. Long API specs and trajectory prefixes limit a single GPU to a few dozen examples per batch. We borrow the fix from B3, which groups training examples so that small batches are full of informative negatives, and replace its semantic similarity graph with a graph of observed harm.

Core Claim

Define the harm of a document $d$ for task $t$ as $h(d,t) = s(t, D^*_t) - s(t, D^*_t \cup \{d\})$. Here $s$ is the state-diff success rate of a fixed labeling agent over a few seeds, and $D^*_t$ is the set of documents for the APIs called in a verified successful trajectory. We claim that a documentation retriever trained contrastively with high-harm documents as negatives will keep agent success close to the oracle-documentation level as the documentation pool grows. We further claim this advantage survives at small batch sizes when batches are assembled from communities in a harm graph. It should beat semantically mined hard negatives and large random batches at matched compute.

What is new is the definition of a negative. B3 builds its negatives from a teacher's embedding ranks and has to guess which near neighbors are false negatives. RAFT samples distractor documents without checking whether they hurt. Here negatives are defined by their effect on execution, and false negatives are ruled out exactly, because $D^*_t$ is known from the verified trajectory.

The claim has a ceiling we must respect. On clean AppWorld, oracle APIs add only 5–10%, because writing the glue code, not finding the API, is the bottleneck. We therefore do not expect large gains in the clean setting. The claim is about robustness as the pool becomes noisier, which is the regime where Agent-Diff observed degradation.

Research Program

The first component is a noise-stressed evaluation built on AppWorld and Agent-Diff. It measures how state-diff success, including AppWorld's collateral-damage tests, decays as the agent's documentation pool grows through four levels. The pool starts as the oracle set. It then adds realistic distractors, meaning real documentation for sibling endpoints within the same app. Next it adds documentation for all apps, which reproduces Agent-Diff's exhaustive setting. Finally it adds an adversarial tier. For that tier we adapt the error-guided tree search from Breaking Focus, which searches for distractions that steer a model toward a specific wrong option. In our version it selects, from the real documentation, the handful of pages that most reduce success. This extends BFCL's binary "should a tool be called" irrelevance checks (BFCL) to graded, multi-step damage.

The second component is harm labeling. For each training task we take the top 50 candidate documents by BM25 keyword match. We measure $h(d,t)$ with an open-weight labeling agent and skip candidates early when a first seed shows no effect, following the pruning idea in Adaptive Distraction. This caps the labeling budget.

The third component is training. A 1–2B text encoder is trained with InfoNCE, the standard contrastive loss that treats retrieval as picking the right document out of a set. Each query is the task instruction plus a short summary of the trajectory so far. Positives are the documents in $D^*_t$ needed at that step. Batches come from METIS partitions of a graph that links tasks sharing high-harm documents, so one task's harmful documents act as in-batch negatives for its neighbors. This is B3's batch assembly with the graph edges changed. At inference the retriever supplies the top-$k$ pages to an unmodified agent, including black-box agents. This matters because PRCA shows that reward-trained filters become tied to a particular generator, so we explicitly test whether harm labels transfer across agents.

Baselines

The naive reference is the agent given the full documentation pool. The file reports it only as a direction of effect: full documentation underperforms relevant documentation, which gives +7.0 pp overall on Agent-Diff and +2.4 on the assertion-weighted score (P=0.73). The upper reference is oracle documentation. For the agent itself we anchor to the reported GPT-4o ReAct scores on AppWorld: 48.8% on Normal and 30.2% on Challenge tasks (AppWorld). We found no reported result on Lacuna for any retrieval-filtered documentation baseline on AppWorld or Agent-Diff, so every retriever baseline is run by us.

The strongest competitor is the same encoder trained with B3's own recipe, using semantic teacher-ranked communities and top-$p$ exclusion at the same batch size. It is the closest method and isolates our key ingredient, harm-defined negatives. It holds B3's reported MMEB results (68.1 avg at 2B), but on a different task. We add a second competitor: the same encoder with random negatives at a 16× larger effective batch, using the memory-saving gradient caching of Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup and matched GPU-hours. A third is a BM25 or off-the-shelf dense retriever. A fourth, for open-weight agents only, is RAFT-style fine-tuning of the agent on distractor-laden contexts; RAFT's own reported figure is 74.00% on HuggingFace APIBench.

We also run two ablations. The first uses harm negatives but random batches, which isolates the batching. The second uses the harm graph but binarized labels, which isolates graded harm.

All retrievers share the data, the encoder, the tuning budget of 20 trials, and top-$k$. Beating a baseline means higher state-diff success at the all-apps and adversarial pool levels, with non-overlapping 95% bootstrap intervals over tasks. Collateral damage must not increase.

Why not just…?

Why not just use a strong off-the-shelf retriever or reranker? Semantic similarity is exactly what makes a sibling endpoint dangerous. In PRCA, off-the-shelf SimCSE and DPR retrieval left enough noise that a trained filter lifted SimCSE+Vicuna by 20%. Our pool-growth curves compare BM25 and dense retrieval against the harm-trained retriever directly. If the off-the-shelf retriever matches it at the all-apps level, the harm labels are not worth their cost, and we will report that.

Why not just use gradient caching with huge random batches? This is the most serious threat. Gradient caching removes the memory limit on batch size, so the batch-size problem would be solved by engineering. B3's evidence is that random negatives mostly contribute near-zero gradient, and its batch-64 model beat LLaVE, which used 4–16× larger batches. That evidence comes from multimodal embedding, however, not documentation retrieval. The matched-compute comparison against a 16× gradient-cached batch settles the question for our setting. If large random batches match harm-community batches, the batching component is dropped, and the contribution reduces to harm-defined negatives.

Why not just fine-tune the agent on distracted contexts (RAFT or DPO)? RAFT is single-turn. DPO on distracted-versus-clean pairs helps static QA (Adaptive Distraction). Neither applies to black-box agents. On an open-weight agent we run RAFT-style training with the same harm-labeled data, which tests directly whether filtering beats teaching the agent to ignore noise. Combining the two is a legitimate outcome.

Why not just prompt the agent to ignore irrelevant docs? Breaking Focus found almost no defensive benefit from such instructions against targeted distraction. We include the prompt as a cheap control at every pool level.

Decisive Checks

Three results would confirm the proposal. First, at the all-apps level on held-out AppWorld apps and on all 224 Agent-Diff tasks, the harm-trained retriever should recover most of the gap between full and oracle documentation. Second, it should beat semantic B3 mining at the same batch size of 32–64. Third, its advantage over semantic mining should widen at the adversarial level. The transfer check labels harm with one open agent and then evaluates GPT-4o and a second family. If gains vanish across agents, harm is agent-specific, reproducing PRCA's coupling problem. Two diagnostics accompany these checks. The first is harm-at-$k$, the fraction of retrieved pages with $h>0$, which shows whether gains come from excluding harmful pages rather than from better recall. The second is the fraction of failures that remain once correct documentation is present, which should match AppWorld's evidence that glue code, not discovery, dominates clean-setting errors.