Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems

Yizhou Peng, Chen Chen, Donghang Wu, Hexin Liu, Eng Siong ChngarXiv 2026 · arXiv

TL;DR

This paper introduces a novel framework for Early Tool Calling (ET) in spoken dialogue systems, allowing agents to trigger external tools mid-utterance as soon as they have "heard enough" to act. By training an omni-modal LLM on a massive new 786-hour synthetic corpus with precise "earliest-actionable-point" labels, the authors demonstrate a mean latency reduction of 1.8–2.0 seconds compared to traditional systems that wait for the user to finish speaking.

What Problem Does This Paper Address?

Current voice assistants (like Siri or Alexa) typically operate on a "turn-end cascade." They wait for a user to stop talking, transcribe the entire utterance, decide which tool to use, extract arguments, and finally execute the call. This sequential process creates a noticeable "dead air" delay of several seconds.

While previous research has looked at incremental speech processing, it hasn't successfully tied it to irreversible external actions like tool calls. The core challenge is a data gap: existing datasets don't mark the Earliest Actionable Point (EAP)—the exact millisecond in an audio stream where a tool and all its required arguments have been grounded. Without this "gold" timing signal, models can't learn when it's safe to fire a call early.

Methods

The authors tackle this with a two-pronged approach: a sophisticated data pipeline and a chunk-based modeling strategy.

1. The Data Pipeline

To create the necessary supervision, they built a synthetic pipeline (see Figure 1) that generates dialogues, synthesizes them into audio, and then uses "gated annotation" to find the EAP.

  • Simulation: LLMs simulate user/agent interactions across 48 domains.
  • Synthesis & Alignment: Text is converted to speech, and word-level timestamps are extracted via forced alignment.
  • Gating: A series of deterministic gates verify that a proposed tool call is executable and that all arguments are actually present in the audio heard so far. This anchors the t_fire label to the audio stream.

![Fig.

  1. Data pipeline overview showing task instantiation, simulation, and gated annotation for early-tool-call events.](/figures/art_be7a8cc2e1b549d98151fab2243cc9ac)

2. Single-Stream Modeling

The model is based on Qwen3-Omni-30B-A3B, an omni-modal LLM adapted via LoRA. It processes audio in 0.6s chunks. After each chunk, the model emits control tokens that represent the "tool state":

  • WAIT: Still listening.
  • UNSURE: Tool identified, but arguments are missing or contested.
  • READY: Fire the tool call now.
  • HOLD: Call sent; continue listening for potential corrections.

The model can also emit a "payload"—either a partial ASR transcript of what was just said or a full JSON tool call.

![Fig.

  1. Single-stream model design interleaving audio chunks with control tokens and payload blocks.](/figures/art_1fb5fa7d3edf4b9ea8d58bc9cf9cbf21)

Datasets

  • Main Corpus: A newly introduced synthetic dataset of 26,694 dialogues (786 hours) across 48 domains. It features a "leakage-free" split where training and test sets have zero overlap in entity values (e.g., specific booking codes).
  • $\tau^2$-bench Rendering: To test generalization, the authors created an audio version of the existing $\tau^2$-bench (a tool-calling benchmark), totaling 4.9 hours of audio.

Key Insights and Ideas

  • Acting vs. Thinking: The paper shifts the focus from "thinking while listening" (internal state) to "acting while listening" (external impact).
  • The "Payload" Trade-off: Interestingly, having the model emit a partial ASR transcript for an external, larger "formatter" LLM to process into JSON works significantly better than having the model generate the JSON itself. The specialized formatter reached an Exact Match (EM) score of 0.536, compared to just 0.173 for the model's self-emitted JSON.
  • Precision is the Bottleneck: While the model is good at finding the right moment to fire (recall ~0.5), it suffers from "trigger-happiness." About 59% of early fires were spurious, meaning the model often thinks it has enough info before it actually does.
  • Lateness Tolerance: Many "errors" in timing are just one chunk (0.6s) late. Allowing for this small delay significantly boosts the F1 scores for the READY state.

Key Results

  • Latency Savings: For correct calls, the model fires 1.8–2.0 seconds earlier than a standard turn-end system in-domain, and 1.1 seconds earlier on the external $\tau^2$-bench.
  • Early Firing Distribution: As shown in Figure 3, roughly half of the correct calls fire before the user finishes their turn. Among these early fires, the median time saved is a substantial 2.6 seconds.
  • Decision Quality: The model achieved a tool-state macro-F1 of 0.48 on held-out data and 0.47 on the external benchmark, showing surprisingly robust transfer from synthetic training to a different task schema.

![Fig.

  1. Distribution of time saved for correct early calls compared to a turn-end baseline.](/figures/art_2706309d1e734c0ba2c5bdc7ce5f3478)

Limitations and Future Work

  • Spurious Fires: The high rate of false-positive READY signals is the primary hurdle for deployment. An agent that acts on incorrect information mid-sentence could be more frustrating than a slow one.
  • Synthetic Data Reliance: The model was trained entirely on synthetic speech. While it generalized to the $\tau^2$-bench rendering, performance on real-world, messy human speech remains an open question.
  • Single-Seed Results: The authors note the results are from a single training run; more extensive hyperparameter tuning and multi-seed validation are needed.
  • Future Direction: The authors suggest that improving "fire precision" is the most critical next step for making early tool calling viable in production environments.

Summary updated

Abstract

Latency-sensitive spoken agents must know not only which tool to call, but also when they have heard enough to act. Waiting for the turn to complete delays execution even when the request is already actionable. However, neither existing spoken agents nor spoken dialogue corpora define the earliest point at which a request becomes actionable. We introduce a synthetic spoken-dialogue pipeline that provides earliest-actionable-point supervision, marking when the observed speech grounds a tool and its required arguments. Through controlled task generation, speech synthesis and alignment, and gated annotation, we construct 26,694 dialogues totaling 786 hours across 48 domains, with disjoint training and test argument values. We additionally create an audio rendering of $τ^2$-bench for external evaluation. We train an omni-modal language model on this corpus, enabling causal, chunk-level decisions and partial-transcript generation for downstream call completion. The model achieves tool-state macro-F1 scores of 0.48 and 0.47 on held-out dialogues and the external benchmark, respectively. For correct calls, the estimated mean lead over a turn-end cascade, using the same formatter, is 1.8-2.0 seconds in-domain and 1.1 seconds on $τ^2$-bench. These results demonstrate a data-driven path toward spoken agents that act while listening. We will open-source the pipeline and the training scripts.
BibTeX
@misc{peng2026heard,
  author = {Yizhou Peng and Chen Chen and Donghang Wu and Hexin Liu and Eng Siong Chng},
  title = {{Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems}},
  year = {2026},
  eprint = {2610.34653},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34653},
}
Figure