Knowing When a Spoken Tool Call Is Ready: Source-Resolved Variability Quenching in Factorized Speech Agents

When someone speaks a command to a voice agent, such as "move my 3pm with Dana to Thursday and email her the new invite," the agent must decide at some moment to commit. It has to fire a tool call with concrete arguments: which meeting, which Dana, which day. Today that moment is picked crudely, usually at the end of the utterance or when a confidence score crosses a threshold. A misheard name then flows into every later tool in the chain. Neuroscience offers a sharper criterion. Before a monkey moves, the trial-to-trial scatter of its motor-cortex activity shrinks, and how far it has shrunk predicts reaction time even when average firing rates do not change. This proposal borrows that idea as an engineering test. Run the speech-to-argument computation several times under small perturbations, and commit only once the internal representation of the argument has stopped wandering.

The new ingredient is to split that scatter by source. A factorized speech recognizer keeps acoustic alignment and linguistic prediction in separate modules. Because of that separation, we can ask whether the argument is unstable because the audio is ambiguous (the right move is to keep listening or ask the user to repeat) or because the language side is unsure (the right move is to ask a clarifying question or refuse the call). We test this on spoken versions of multi-step tool benchmarks. The decision rule runs on a real edge board, not in simulation, because the extra forward passes cost wall-clock time and energy that simulation hides.

Core Claim

Across-trial variance of the argument representation, split into an acoustic part and a linguistic part, predicts tool-argument errors better than single-pass confidence does. Routing on the dominant source (commit, wait, clarify, or reject) should raise multi-step task success on spoken inputs at matched latency on real hardware. The claim is falsifiable at three points. The variance may add no information beyond log-probability. The source split may not match the ground-truth cause of an error. Or the gains may disappear once the device's real latency is charged.

Grounding in Prior Work

Three neuroscience results motivate using variance as a readiness signal. Yu et al. 2006 showed that premotor variability at the go cue predicts reaction time. Moore 2009 showed that the drop in the Fano factor (variance divided by mean spike count) before saccades predicts reaction time even with no change in mean rate. This second result is the empirical basis for expecting variance to carry information that a mean-confidence score lacks. Schimel et al. 2024 modeled cortex as a recurrent network under optimal control and found that compound movements need a separate preparation phase before each segment. We copy that structure by re-running the readiness check before every tool call, after each upstream tool returns. None of these papers touch discrete tokens or tools, so the translation is a hypothesis to test, not an inherited result.

The speech side supplies the mechanism. The Factorized Neural Transducer splits the decoder into two parts. A blank predictor handles timing, deciding when to emit nothing. A vocabulary predictor acts as a standalone language model. The paper uses this split for text-only domain adaptation, reporting 15.4–19.4% relative WER gains on LibriSpeech test-other. It also pays a "factorization tax" of 8.32% versus 8.10% unadapted WER and 103M versus 84M parameters. Hybrid Autoregressive Transducer (HAT) separates blank emission from label emission so that the internal language prior can be subtracted exactly. It reports 6.6% versus 8.0% WER on voice search. Both papers treat factorization as a way to adapt the recognizer. We instead use the split as a diagnostic: it lets us perturb the acoustic path and the linguistic path independently and attribute the resulting uncertainty to each. Neither paper connects its recognizer to tools.

The tool-use literature shows where the errors hurt most. ToolQA attributes nearly 50% of agent errors to argument mistakes and reports accuracy falling from 43.1% on single-tool questions to 8.2% on questions that compose tools. OrchestrationBench finds that agents force tool calls when they should reject them. SLURP quantifies how a speech cascade degrades understanding: action accuracy falls from 86.99% on gold text to 81.42% on ASR output, and SLU-F1 from 78.19 to 70.84. Learning Compositional Tasks via Trigger Compositions constrains our design. It found that intermediate work placed after the answer collapses accuracy to about 30%, while the same work placed before the answer reaches 98.6%. That is why our check runs before commit and not as a post-hoc verifier. We note the result comes from synthetic arithmetic and may transfer only loosely.

The hardware lens comes from Air Learning. It showed that policies tuned at desktop latency (11 ms) break on a Raspberry Pi (up to 400 ms), and that injecting profiled latency distributions into training closes the flight-time gap from 37.73% to 0.5%. When Simulation Lies shows that tool agents lose about 30% accuracy under runtime transition faults. Its domain-randomized RL recipe closes about 27% of that gap, but it studies single-turn calls only. Our spoken, multi-step, on-device setting sits outside both papers.

Research Program

The front end is a factorized transducer. It is trained on public speech, then adapted text-only to each tool domain's entity catalogs, following the Factorized Neural Transducer recipe. Its output is streamed into an open agent model that emits tool calls. We use gpt-oss-20b, which scores 0.654 on the MCP-Bench leaderboard. At each candidate commit point $t$, meaning the moment the agent's pending tool call has all argument slots filled, we run $K$ perturbed "trials" and record the argument representation $h_k(t)$. This is the agent's hidden state pooled over the argument span.

The acoustic trials hold the linguistic path fixed and perturb the audio: additive noise at the measured device noise floor, small speed and time shifts, and encoder dropout. The linguistic trials hold the audio fixed and perturb the vocabulary predictor: dropout, and swapping between the general and domain-adapted language models. We define a quenching index for each source $s$:

$$Q_s(t) = \frac{\operatorname{tr}\,\mathrm{Cov}_k[h_k^{(s)}(t)]}{\operatorname{tr}\,\mathrm{Cov}_k[h_k^{(s)}(t_0)]}$$

Here $h_k^{(s)}$ is the representation under trial $k$ of source $s$. The time $t_0$ is the first point at which the slot was filled, which plays the role of the "unprepared" baseline. Normalizing by $t_0$ follows the Fano-factor logic of measuring a drop relative to the unprepared state, not an absolute level.

The routing rule is simple. The agent commits when both indices stay below a threshold for $m$ consecutive frames. It waits for more audio, or asks the user to repeat, when $Q_{\text{ac}}$ dominates. It asks a clarifying question, or rejects the call, when $Q_{\text{ling}}$ dominates and no further audio is expected. After each tool returns, the index resets for the next call. Thresholds are tuned on development splits only.

Everything runs on an Arm big.LITTLE board of the ODROID-XU3 class. We report measured latency and energy, with power estimated using the validated modeling approach of Accurate and stable empirical CPU power modelling, which reports 2.8% MAPE. The agent model runs off-board. Its measured round-trip latency distribution is injected into the multi-turn loop in the Air Learning manner. Transition faults drawn from RobustBench-TC's categories are added as well.

Benchmarks

No spoken multi-step tool benchmark exists in the literature available to us, so we build one. We take the OrchestrationBench English subset (219 sessions, 706 tool calls) and the multi-tool ToolQA hard questions. Every user turn is synthesized with several TTS voices and noise styles. A subset of about 1,000 turns is also recorded by human speakers. The TTS-versus-human difference is itself a gap between simulated and real inputs, and we report it. SLURP supplies real spoken commands for single-step argument extraction, where published numbers exist.

Metrics are chosen to match each failure type. For arguments we use exact-match accuracy and SLU-F1. For tasks we use OrchestrationBench's execution and planning scores. For the reject problem we use correct-reject rate. For cost we report commit latency and energy measured on the device. Results are also reported as error-versus-latency curves, so that a method that waits longer cannot win just by waiting.

Baselines

The naive reference is a plain cascade. The same factorized recognizer transcribes to the utterance endpoint, and the agent commits its tool call immediately. It gets the same text-only domain adaptation, the same agent, and the same device. Every other method must beat this cascade at equal or lower device latency.

The strongest single-pass competitor is a confidence gate. It commits when the argument tokens' log-probability, corrected with HAT-style internal language model subtraction, clears a tuned threshold. Low-confidence calls are routed to clarification. This is the most dangerous baseline because it costs one forward pass instead of $K$. Moore's result is exactly a claim that variance adds information beyond the mean, so this is the right contest. The confidence gate gets the same development-set threshold search as our method. Beating it means lower argument error at matched commit latency on the device, plus a statistically reliable gain in AUROC for error prediction when variance is added to log-probability in a logistic model.

The second competitor is output-level self-consistency. It takes the same $K$ perturbed trials and majority-votes on the decoded argument strings, committing only when the vote is unanimous or above a threshold. It uses the same compute as our method but ignores both hidden-state geometry and source attribution. Beating it requires better routing decisions, not just better detection of errors.

The third competitor is training-time robustness. Following the domain-randomization recipe of When Simulation Lies, we fine-tune the agent on perturbed transcripts and injected transition faults, using a training budget matched to our threshold tuning plus the agent size. For reference, the paper reports that a 3B model retains about 75% of its clean accuracy and closes about 27% of the transition gap. No reported result was found on Lacuna for any method on spoken OrchestrationBench or spoken ToolQA. On the text side, the reported numbers give upper references: OrchestrationBench tool execution of 0.8821 for Claude 3.5 Sonnet and planning of 0.8504 for Gemini 2.5 Pro. On real speech, SLURP's HerMiT reaches 81.42% action accuracy on ASR output.

Two ablations isolate the key ingredient. The first uses pooled variance without the source split, which tests whether attribution matters. The second applies the same trial procedure to a monolithic RNN-T of matched size, where acoustic and linguistic perturbations cannot be cleanly separated. That tests whether factorization is needed or merely convenient.

Why not just…?

Why not just threshold the model's confidence? This is the real threat, and it is settled directly. If adding $Q_{\text{ac}}$ and $Q_{\text{ling}}$ to log-probability does not improve error-prediction AUROC on held-out human-recorded speech, the variance signal is redundant and we drop it. The only argument in its favor is indirect. Moore found that variance predicts behavior when the mean does not. Separately, internal language model overconfidence is well documented in transducers, which is the motivation behind HAT and internal language model estimation. A confidence score also cannot say why the model is uncertain, and the "why" is what decides between waiting and clarifying.

Why not just sample several outputs and vote? Voting detects instability but cannot attribute it to a source. Our routing experiment tests whether attribution buys anything. We inject pure acoustic corruption, such as noise over an entity, and pure linguistic novelty, such as an out-of-catalog but cleanly spoken name. We then check whether each method chooses the right action: wait or repeat for the first, clarify or reject for the second. If voting plus a fixed "ask the user" policy matches our success rate and latency, the source split is not worth its cost. We will say so.

Why not just train the agent to be robust instead? Domain randomization changes the policy but gives no signal at inference time for when to hold back. The problem we target, forced calls that should have been rejected, is a failure of that inference-time decision. Its reported transition-gap closure of about 27% also leaves most of the gap open. The two methods can be combined, and we report our gate on top of the randomized agent to check whether the gains stack.

Why not just use a verifier after the call is drafted? In a tool pipeline, a committed call has side effects, such as an email sent or a meeting moved. The trigger-composition study also found that post-answer checking recovers far less than pre-answer work, though that evidence comes from synthetic arithmetic. We include a draft-then-verify baseline at equal compute to test this in the tool setting.

Decisive Checks

Four results decide whether the idea holds. First, variance must beat confidence at error prediction on human-recorded speech. This result gates the whole project. Second, in the controlled injection study, the source split must agree with the known cause of each error well above chance. Third, the routed agent must dominate the cascade and the confidence gate on the error-versus-latency curve measured on the ARM board, not merely on FLOP counts. We will report where a desktop-timed evaluation ranks the methods differently from the device, since that is the gap between simulation and hardware this proposal exists to expose. Fourth, following Schimel's segmented-preparation result, the per-step reset must beat a single check at the start of each session on sessions with three or more tool calls, where errors from early calls have the most room to cascade.