Teaching Small Tool-Use Agents to Translate Before They Act

Customer-service agents built on language models fail in a specific, repeatable way: the user says "my card got charged twice for the blue sweater," and the agent must turn that into an order lookup, a specific item ID, a refund tool, and a store policy clause, none of which share words with what the user said. This is the vocabulary mismatch, or lexical gap, between how people talk and how tools and databases are named. Large models handle it tolerably when several prompted copies of the model split the work. One copy pulls out the applicable rules, another shortlists the right tools, another asks clarifying questions. Small open models cannot run that kind of multi-agent protocol, and training them with reinforcement learning (trial-and-error optimization against a success signal) from scratch collapses because they rarely produce a valid tool call to learn from.

We propose to compress a multi-agent teacher's division of labor into a single small model by giving the student one explicit intermediate step. Before every tool call, the student writes a short "grounding" segment that restates the user's request in the environment's own vocabulary: the tool name, the argument values, and the policy clause that applies. A brief supervised warm-start on synthetic multi-agent trajectories seeds this segment. Reinforcement learning then refines it, with a cheap, checkable reward on whether the grounding segment names the right tool and arguments. The claim to test is narrow: an explicit, rewarded translation step is what lets small agents close the lexical gap, and its benefit should grow as user phrasing drifts further from the tool vocabulary.

Core Claim

Prior work supplies three pieces but never puts them together. Satori shows that reinforcement learning cannot teach a model to emit new structural actions from scratch. It uses a Generator–Critic–Reward-model system to synthesize about 10K warm-start trajectories that seed meta-action tokens, then runs RL. That work is confined to math, where there is no gap between problem vocabulary and solution vocabulary. IRMA shows that decomposing input handling into a constraints agent, a tool-selector agent, and clarifying questions improves reliability on τ-bench. It does this only by prompting GPT-4o, with several model calls per turn and no training. RaFe trains a small query rewriter with a supervised cold start followed by preference optimization against a cheap reranker reward. It handles retrieval queries only, with a single rewriting step and no tools or policies.

Our proposal treats IRMA-style decomposition as a source of training data rather than an inference-time architecture. It treats lexical grounding as a learned intermediate action in the Satori sense, and it rewards that action with a verifiable signal in the RaFe sense. The thing that would be new is evidence that this intermediate step, rather than distillation in general, carries the gain on lexical-gap cases.

Research Program

Synthetic task generation. Training tasks are synthesized, not drawn from τ-bench. The benchmark is small: τ-bench has 115 retail and 50 airline tasks, so it must be held out entirely. Following the grounded-seeding idea of Magicoder, which seeds a teacher with real code snippets to avoid narrow synthetic data, we seed task generation with real database records and tool schemas from the τ-bench and τ²-bench environments. A teacher writes a target action sequence and a user persona. A separate paraphrasing pass then rewrites the persona's instructions at controlled levels of lexical overlap with tool and argument names. Overlap is measured as token overlap between user instructions and the gold tool calls. This gives us a dial on the size of the lexical gap.

Teacher trajectories. A frontier model runs an IRMA-style decomposition on these tasks. Separate prompted agents extract constraints, shortlist tools, and decide whether to ask a clarifying question. We then collapse each turn's multi-agent output into a single student-format turn: a grounding segment, followed by either a question or a tool call. Following the filtering practice of Referee and Satori, only trajectories that reach the target database state are kept.

Warm-start. The student is Qwen3-8B, the scale UserRL trained. That work found a supervised cold start is required before multi-turn GRPO (group relative policy optimization, an RL method that compares sampled responses within a batch). Following Balancing the Budget, whose more than 1,000 runs found that 10–25% of a fixed budget spent on supervised data before preference optimization is near Pareto-optimal, we sweep the supervised share at 5%, 10%, 25%, and 50% of a fixed trajectory budget.

RL stage. The RL stage uses GRPO against a user simulator with reward

$$r = r_{\text{outcome}} + \lambda\, r_{\text{ground}},$$

The two terms are defined as follows. $r_{\text{outcome}}$ is 1 when the final database state matches the target and 0 otherwise. $r_{\text{ground}}$ is the fraction of gold tool names and argument values that appear in the grounding segment before the corresponding call. $\lambda$ is a weight we anneal to zero. The dense term exists because sparse terminal rewards cause training collapse, as reported in Efficient Reasoning via Reward Model. Annealing it guards against the agent learning to write a good grounding segment and then ignoring it.

Baselines

The naive reference point is Qwen3-8B with standard function calling and no task-specific training. For it and for small-model τ-bench numbers in general, we found no reported result on Lacuna for an 8B model. The file says only that Llama-3-70B "struggled to clear even 20% success on harder tasks" on τ-bench.

The strongest prompted reference is IRMA run with GPT-4o. It reports a pass^5 (all five repeated attempts succeed) of around 43% on τ-bench, and beats standard function calling by 12.7% in pass^5 (IRMA). The plain GPT-4o function-calling agent reports 61.2% pass^1 on retail and 35.2% on airline (τ-bench). IRMA is the strongest competitor because it addresses the same failure mode with the same decomposition. Beating it with an 8B model at a fraction of the inference calls is the headline target. Matching it within a few points of pass^5 at one call per turn would already be a meaningful result.

The strongest trained competitor is a UserRL-style pipeline. It uses supervised cold start on the same filtered teacher trajectories with the grounding segments removed, then GRPO on outcome reward only. It gets the identical task pool, trajectory count, token budget, RL steps, and hyperparameter search budget. This is also the key ablation, because it isolates the grounding segment while holding data and compute fixed. A second ablation keeps the grounding segment but sets $\lambda=0$, which separates the value of the format from the value of the dense reward.

Finally, we compare against AgentVocab, which expands the tokenizer with schema tokens mined from tool traces and fine-tunes with supervised learning. It reports 22.1% fewer input tokens and 17.7% lower latency on τ-bench at competitive accuracy, but underperforms vanilla supervised fine-tuning on τ²-bench Telecom. Absolute accuracy numbers are not in the file, so we reproduce it on our data.

Beating a baseline means a pass^5 gain whose confidence interval over three seeds excludes zero, on the held-out τ-bench and τ²-bench tasks.

Why not just…?

Why not just distill a monolithic frontier agent's trajectories into the small model? This is the most serious threat. If teacher quality is all that matters, the grounding segment is decoration. The stripped-segment ablation above is the direct test, run with identical trajectories and identical token counts. We will also run a version where the stripped tokens are replaced by teacher chain-of-thought of equal length, so that "more reasoning tokens" is not confounded with "schema-vocabulary tokens." If grounding shows no advantage, especially on the high-gap split, we would drop the central claim and report that decomposition helps only as a data-quality filter.

Why not just run IRMA's multi-agent prompting on the small model? The cited multi-agent papers report that models under about 13B cannot sustain separate personas and protocols (Self-collaboration Code Generation, MACM). Even if it worked, it would multiply inference calls per turn. We include IRMA-on-Qwen3-8B as a measured baseline rather than assuming it fails.

Why not just fix the vocabulary at the tokenizer level, as AgentVocab does? AgentVocab makes schema strings cheaper to read and write. It does not teach the mapping from "charged twice" to a refund tool and a policy clause, and its Telecom regression suggests compression alone can cost accuracy. We expect the two to be complementary, and we test the combination as one extra run.

Why not just ask clarifying questions more often? IRMA's follow-up-question strategy resolves missing information, but it cannot resolve a mismatch the agent does not notice. Our logs will separate failures into wrong-tool, wrong-argument, and policy-violation categories, so we can see which ones questions fix and which ones grounding fixes.

Proposed Experiments

The evaluation sets are τ-bench retail and airline and τ²-bench, including Telecom. The metrics are pass^1, pass^5, turns per success, and inference tokens. The first decisive check is the lexical-gap curve. We paraphrase each held-out task's user instructions into low, medium, and high gap versions with the same procedure used in training, and plot success against gap level for the full method and the stripped-segment ablation. The proposal predicts that the gap between the two curves widens as lexical overlap falls. A flat difference would mean the gain is not about vocabulary at all.

The second check is a causal intervention at test time. We corrupt the student's grounding segment, for example by swapping one argument value for a plausible alternative, and measure how often the subsequent tool call follows the corruption. This shows whether the policy actually conditions on its translation. The third check is the supervised-share sweep. If the optimum lands in the 10–25% range found by Kang et al., the warm-start mechanism generalizes from format anchoring to a semantic intermediate step. If it shifts sharply upward, grounding is behaving more like content that must be imitated than like a format that RL can refine.