Voice agents that book flights or move money fail in a characteristic way. They mishear one thing, such as a digit in an account number, and then act on it with confidence. The wrong value becomes a wrong tool argument, the tool errors out, and the agent retries until its budget runs out. When voice agents are trained with reinforcement learning (RL, learning from trial-and-error reward), the moments that matter most are the few turns right after a mishearing, where the agent could confirm, read back, or ask again. These moments are rare, buried deep inside long conversations, and usually end in failure. As a result, the training signal for recovery is weak.
We propose to stop waiting for the agent to stumble into these moments. The training simulator knows what the user actually said, so it can find the exact turn where the agent's understanding first diverged from the truth. It can save that conversation prefix and later restart many practice attempts from it. The idea comes from uncertainty-aware exploration, which means spending experience where the learner is least sure of the outcome. We apply it to the choice of which saved mishearing states to replay: we favor those where the agent's chance of recovering is still uncertain and not yet near zero or one. The claim is that aiming exploration at the states where speech-recognition errors enter the conversation will teach error recovery more efficiently than adding optimism to what the agent generates. It should also beat the hand-shaped per-turn rewards used today.
Why this gap is real
The motivating evidence comes from two voice-agent studies. SpeechGym builds a fully local environment in which two omni-modal models (Qwen3-Omni-30B) talk in audio while tool calls stay textual. It reports that perceptual mishearing rises from 2% of turns in text to 32% in audio. It trains with Group Relative Policy Optimization (GRPO), which scores each sampled rollout against the other rollouts in its group. To avoid near-zero gradients when almost every rollout fails, it adds hand-crafted per-turn rewards for successful tool calls and penalties for failed ones. This raises τ-Voice pass@1 from 24% to 53% and the recovery rate after mishearing from 42% to 62%. More than a third of misheard conversations still do not recover.
τ-Voice sets the scale of the problem. Task completion is 85% for a text GPT-5 agent but only 26–38% for voice agents under realistic audio. It also attributes 79–90% of failures to the agent's behavior rather than to audio quality. So the bottleneck is what the agent does after perception goes wrong, which is a sequential decision problem and not a transcription problem.
The exploration literature supplies the conceptual tool and one constraint. Online Apprenticeship Learning and OGAPI show that an agent imitating a flawless expert never learns recovery. They fix this with optimistic bonuses that push the agent into states outside the expert's path. Their guarantees hold in continuous control and linear or tabular models. We keep their insight, that recovery requires visiting off-path states, but reach those states by resetting a simulator we control rather than by bonus-driven wandering.
General Exploratory Bonus (GEB) shows that in KL-regularized language-model RL, exploration bonuses get pulled back toward responses the reference model already favors, which the authors call a "gravity well." It proposes reference-dependent bonuses that restore exploration of low-probability text. This result acts as a constraint on our design. Bonuses in token space fight the regularizer, whereas choosing start states leaves the per-token objective and KL term unchanged. The regularizer then has nothing to cancel.
Speech-error work on the other side offers only test-time or static remedies. Contextual ASR error handling re-ranks N-best speech-recognizer hypotheses with dialogue context and phonetic matching. Confidence-feature NER feeds recognizer confidence into a tagger to suppress false entities. Both repair or down-weight the transcript. Neither teaches a policy what to do across later turns once an error has slipped through.
Core Claim
In voice-agent RL, uncertainty should direct exploration over where conversations restart: specifically, over the states produced by perceptual errors. It should not act as an optimism bonus over the agent's tokens, and it should not act as optimism about what the user said. Optimism about the user's words is exactly the "confidently wrong over-action" that τ-Voice documents.
Concretely, we expect restarts drawn from a buffer of mishearing states, allocated by an upper-confidence rule over recovery rates, to beat SpeechGym's reward-shaped GRPO at matched compute. We expect gains on held-out τ-Voice pass@1, on post-mishearing recovery rate, and on repeated-trial consistency.
Research Program
Locating branch points. After every training rollout in SpeechGym, we compare the agent's slot values against the user simulator's ground-truth task specification. We check the values in its text reasoning and in its tool arguments. The first turn where a slot diverges is marked as a branch point. Because tool calls are textual, the divergence can be attributed to perception and not to planning. We store three things at each branch point: the audio-and-text prefix, the database state, and the user simulator's state.
Grouping branch points. Each branch point is assigned to a cell defined by domain, slot type (digit strings, names, dates, amounts), acoustic condition (accent, noise type), and turn-depth bucket. The high-stakes Banking domain from SpeechGym, where single-digit errors are fatal, contributes many digit-string cells.
Scoring cells. For each cell $c$ we track the current policy's recovery rate $\hat p_c$ and the number of attempts $n_c$. Here "recovery" means task success after restarting from a stored prefix. Counts decay geometrically so the estimates follow the changing policy. Each cell gets the score
$$s_c = \hat p_c (1-\hat p_c) + \beta \sqrt{\frac{\log N}{n_c + 1}},$$
where $N$ is the total number of restart attempts so far and $\beta$ scales the exploration term. The first term is the outcome variance of a group of restarts. GRPO's gradient vanishes when a group's outcomes are all equal, so this variance measures how much a group can teach. The second term is a standard confidence width that keeps rarely-tried cells from being ignored.
Training. Each batch spends a fraction $\lambda$ of its rollout budget on restart groups. These groups draw prefixes from the highest-scoring cells, and all $G$ samples in a group share the same misheard prefix. Everything after the prefix is generated by the current policy, so updates remain on-policy from that point. Advantages then compare recovery strategies from an identical broken state. The rest of the batch uses ordinary from-scratch rollouts, so the agent still learns to avoid mishearing situations in the first place. Prefixes older than a fixed number of policy updates are evicted, which limits training on states the current policy would never reach.
Analysis. Beyond the headline scores, we measure the frequency of read-back or confirmation before irreversible writes. We also measure unnecessary clarification turns, dead-loop rate, and unauthorized write rate.
Baselines
The naive reference point is the untrained Qwen3-Omni-30B agent. SpeechGym reports 24% pass@1 on τ-Voice for this model. We re-measure it in our harness so all comparisons share one evaluator.
The strongest existing method is SpeechGym's GRPO with process reward shaping. It is the only published RL method trained end-to-end on audio-native voice agents, and it reports 53% τ-Voice pass@1 and a post-mishearing recovery rate of 62% (up from 42%). We re-run it from the same checkpoint with the same training domains (audio-adapted τ²-bench plus Banking) and the same GRPO group size and KL coefficient. We match compute by total generated tokens and audio seconds, not by episode count, because restart episodes are shorter. It receives the same hyperparameter search budget as our method. Beating it means a higher τ-Voice pass@1 and recovery rate than our reproduction, with the gap exceeding seed-to-seed spread over at least three seeds. It also means no loss on pass^k consistency.
The first ablation, uniform restarts, draws prefixes uniformly from the same branch-point buffer. It isolates the restart mechanism from the uncertainty-based allocation.
The second ablation, variance-only, sets $\beta = 0 $. It isolates the confidence-width term against a plain "train where success is near 50%" curriculum.
The third comparison adds a GEB bonus (General Exploratory Bonus) to SpeechGym's GRPO. It tests whether token-space optimism achieves the same effect, and it gets its bonus scale tuned within the same budget.
The fourth comparison is matched acoustic augmentation. It raises the rate of noisy and accented user speech in from-scratch rollouts until the number of mishearing events matches what our restarts supply. This tests whether we are simply delivering more errors.
Why not just…?
Why not just keep SpeechGym's process reward shaping and train longer? This is the most serious competitor. A 24%→53% jump is large, and longer training might close much of the remaining gap. However, the shaping rewards tool-call success. After a mishearing, a tool call with a wrong digit can still succeed, for example by querying the wrong account. The shaping therefore does not penalize the behavior that causes over-action, and it gives no credit to a confirmation question that makes no tool call at all. The matched-compute learning curves settle this question. If SpeechGym's curve reaches our final recovery rate within, say, 1.5× our compute, the restart idea is not worth its engineering cost, and we will report that.
Why not just use a token-level exploration bonus like GEB? GEB was built for the case where the KL term pulls exploration back toward familiar text, and it helps on text reasoning (General Exploratory Bonus). But it explores whatever the agent happens to be generating in its rollouts. If the agent rarely reaches a post-mishearing state, token novelty there is rarely sampled. The GEB-augmented baseline measures this directly, and a breakdown of results by cell shows whether its gains concentrate on recovery.
Why not just inject more acoustic noise, or restart uniformly? Noise augmentation can raise the number of errors seen, and uniform restarts may capture most of the benefit. The matched-augmentation and uniform-restart ablations test these two possibilities separately. If uniform restarts match the uncertainty-directed allocation, we will drop the uncertainty claim and present restart-from-mishearing as the contribution. The data-augmentation evidence also sets a limit here. τ-bench shows agents at 60% pass^1 falling below 25% at pass^8, which suggests exposure alone does not produce reliable sequential behavior.
Why not just repair transcripts at inference time? N-best re-ranking improves slot recall (Contextual ASR error handling). But the audio-native agent produces no N-best list, and τ-Voice attributes most failures to behavior rather than transcription (τ-Voice). Inference-time repair also cannot decide whether to confirm before an irreversible write. We treat it as complementary rather than competing.
Proposed Experiments
All training uses SpeechGym with the audio-adapted τ²-Bench domains and the Banking domain. Held-out evaluation uses τ-Voice's 278 tasks under clean and realistic conditions, and reports pass@1, pass^k, recovery rate after mishearing, turns, and unauthorized writes. Banking results are reported separately, because SpeechGym does not break them out and we could find no reported per-domain number for it on Lacuna.
The first decisive check asks whether restarts beat reproduced SpeechGym at matched compute on τ-Voice realistic-condition pass@1 and recovery rate.
The second asks whether uncertainty-directed allocation beats both uniform restarts and $\beta = 0$. We would also expect its advantage to be largest early in training, when most cells have few attempts.
The third tests generalization. We hold out entire accent conditions from the training cells and evaluate on τ-Voice's accented subsets, where the benchmark reports an average drop of 10 percentage points. This checks whether recovery learned from some mishearings transfers to others rather than overfitting to stored prefixes.
The fourth is behavioral. Gains should come with more confirmations before irreversible actions and fewer dead loops, without a large rise in total turns. A policy that reaches higher pass@1 by asking the user to repeat everything would count as a failure of the method, not a success.
References
- SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning
- $\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains
- Online Apprenticeship Learning
- Learning from Demonstration: Provably Efficient Adversarial Policy Imitation with Linear Function Approximation
- General Exploratory Bonus for Optimistic Exploration in RLHF
- Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI
- Incorporating Speech Recognition Confidence into Discriminative Named Entity Recognition of Speech Data
- {$\tau$}-bench: A Benchmark for \underline{T}ool-\underline{A}gent-\underline{U}ser Interaction in Real-World Domains
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment