TL;DR
This paper introduces a novel framework for Early Tool Calling (ET) in spoken dialogue systems, allowing agents to trigger external tools mid-utterance as soon as they have "heard enough" to act. By training an omni-modal LLM on a massive new 786-hour synthetic corpus with precise "earliest-actionable-point" labels, the authors demonstrate a mean latency reduction of 1.8–2.0 seconds compared to traditional systems that wait for the user to finish speaking.
What Problem Does This Paper Address?
Current voice assistants (like Siri or Alexa) typically operate on a "turn-end cascade." They wait for a user to stop talking, transcribe the entire utterance, decide which tool to use, extract arguments, and finally execute the call. This sequential process creates a noticeable "dead air" delay of several seconds.
While previous research has looked at incremental speech processing, it hasn't successfully tied it to irreversible external actions like tool calls. The core challenge is a data gap: existing datasets don't mark the Earliest Actionable Point (EAP)—the exact millisecond in an audio stream where a tool and all its required arguments have been grounded. Without this "gold" timing signal, models can't learn when it's safe to fire a call early.
Methods
The authors tackle this with a two-pronged approach: a sophisticated data pipeline and a chunk-based modeling strategy.
1. The Data Pipeline
To create the necessary supervision, they built a synthetic pipeline (see Figure 1) that generates dialogues, synthesizes them into audio, and then uses "gated annotation" to find the EAP.
- Simulation: LLMs simulate user/agent interactions across 48 domains.
- Synthesis & Alignment: Text is converted to speech, and word-level timestamps are extracted via forced alignment.
- Gating: A series of deterministic gates verify that a proposed tool call is executable and that all arguments are actually present in the audio heard so far. This anchors the
t_firelabel to the audio stream.

2. Single-Stream Modeling
The model is based on Qwen3-Omni-30B-A3B, an omni-modal LLM adapted via LoRA. It processes audio in 0.6s chunks. After each chunk, the model emits control tokens that represent the "tool state":
WAIT: Still listening.UNSURE: Tool identified, but arguments are missing or contested.READY: Fire the tool call now.HOLD: Call sent; continue listening for potential corrections.
The model can also emit a "payload"—either a partial ASR transcript of what was just said or a full JSON tool call.

Datasets
- Main Corpus: A newly introduced synthetic dataset of 26,694 dialogues (786 hours) across 48 domains. It features a "leakage-free" split where training and test sets have zero overlap in entity values (e.g., specific booking codes).
- $\tau^2$-bench Rendering: To test generalization, the authors created an audio version of the existing $\tau^2$-bench (a tool-calling benchmark), totaling 4.9 hours of audio.
Key Insights and Ideas
- Acting vs. Thinking: The paper shifts the focus from "thinking while listening" (internal state) to "acting while listening" (external impact).
- The "Payload" Trade-off: Interestingly, having the model emit a partial ASR transcript for an external, larger "formatter" LLM to process into JSON works significantly better than having the model generate the JSON itself. The specialized formatter reached an Exact Match (EM) score of 0.536, compared to just 0.173 for the model's self-emitted JSON.
- Precision is the Bottleneck: While the model is good at finding the right moment to fire (recall ~0.5), it suffers from "trigger-happiness." About 59% of early fires were spurious, meaning the model often thinks it has enough info before it actually does.
- Lateness Tolerance: Many "errors" in timing are just one chunk (0.6s) late. Allowing for this small delay significantly boosts the F1 scores for the
READYstate.
Key Results
- Latency Savings: For correct calls, the model fires 1.8–2.0 seconds earlier than a standard turn-end system in-domain, and 1.1 seconds earlier on the external $\tau^2$-bench.
- Early Firing Distribution: As shown in Figure 3, roughly half of the correct calls fire before the user finishes their turn. Among these early fires, the median time saved is a substantial 2.6 seconds.
- Decision Quality: The model achieved a tool-state macro-F1 of 0.48 on held-out data and 0.47 on the external benchmark, showing surprisingly robust transfer from synthetic training to a different task schema.

Limitations and Future Work
- Spurious Fires: The high rate of false-positive
READYsignals is the primary hurdle for deployment. An agent that acts on incorrect information mid-sentence could be more frustrating than a slow one. - Synthetic Data Reliance: The model was trained entirely on synthetic speech. While it generalized to the $\tau^2$-bench rendering, performance on real-world, messy human speech remains an open question.
- Single-Seed Results: The authors note the results are from a single training run; more extensive hyperparameter tuning and multi-seed validation are needed.
- Future Direction: The authors suggest that improving "fire precision" is the most critical next step for making early tool calling viable in production environments.
Summary updated