VoiceAgentBench: Are Voice Assistants ready for agentic tasks?

Dhruv Jain, Harshit Shukla, Gautam Rajeev, Ashish Kulkarni, Chandra Khatri, Shubham AgarwalSubmission to ICLR 2026 · Withdrawn

TL;DR

While text-based LLM agents are rapidly improving at invoking external APIs and planning complex workflows, voice-based agents remain largely untested on these capabilities. VoiceAgentBench (VAB) introduces the first benchmark to evaluate speech language models (SpeechLMs) and cascaded ASR–LLM pipelines on structured tool use, sequential reasoning, and safety across English, Hindi, and five other Indic languages. The findings show that while end-to-end SpeechLMs offer a massive latency advantage, they lag drastically behind text pipelines in reasoning, cultural grounding, and safety.

What Problem Does This Paper Address?

Voice assistants have evolved from simple voice recorders to multimodal models that directly ingest audio and generate text or speech. However, evaluations of SpeechLMs remain stuck on primitive benchmarks: automated speech recognition (ASR), translation, or single-turn factual question answering.

In contrast, real-world voice agents must act: they need to parse user intent from spoken speech, select and validate external APIs, fill structured arguments, chain interdependent tools (e.g., geolocating an address before ordering a ride), and refuse malicious instructions. Prior tool-use benchmarks were almost entirely text-only, English-centric, and blind to acoustic or cultural nuances. VoiceAgentBench fills this gap by testing whether modern speech-driven models can handle end-to-end agentic workflows.

Methods

  • Layered Evaluation Framework: Instead of simple binary pass/fail checks, the authors evaluate tool usage through progressive criteria:
  • Tool Selection (TS): Regex exact-match on expected API names.
  • Tool Call Structure (TCS): Strict JSON schema and datatype verification using Pydantic models.
  • Parameter Filling (PF): Semantic argument correctness judged by GPT-4o-mini with step-by-step reasoning (validated against human annotators with $\kappa = 0.795$).
  • Refusal Rate (RR): Safety assessment evaluating whether models decline adversarial queries.
  • Acoustic Diversity via Farthest Point Sampling (FPS): To synthesize diverse, natural speech without relying on a single synthetic voice profile, the authors extract speaker embeddings (via ECAPA-TDNN) from open-source Indic speech datasets. Using Farthest Point Sampling (FPS), they select reference audios maximizing acoustic spread across pitch, gender, and accents for TTS voice cloning.
  • Target Systems: Evaluates both end-to-end SpeechLMs (KimiAudio 7B, Qwen2.5-Omni 7B, AudioFlamingo3 7B) and cascaded ASR–LLM pipelines (Whisper v3 Large paired with Qwen3 8B, Gemma3 27B, or Llama 3.3 70B).

Datasets

The authors created VoiceAgentBench (VAB), a dataset of 6,092 synthetic spoken queries across 7 languages (English, Hindi, Bengali, Marathi, Tamil, Telugu, and Malayalam). The benchmark spans six distinct task categories:

  1. Single Tool Parameter Filling (adapted from BFCL; 151 functions)
  2. Single Tool Retrieval + Parameter Filling (from BFCL; 180 functions)
  3. Parallel Tool Invocation (from BFCL; 246 functions)
  4. Sequential Dependent Tool Calling (a newly designed dataset of 21 tools across cab booking, food delivery, and bill payment services where function calls feed into downstream APIs)
  5. Multi-Turn Dialogue Tool Calling (rewritten from API-Bank Level 2)
  6. Harmful Request Refusal (adapted from AgentHarm with Indian-specific contexts)

Key Insights and Ideas

  • The Latency vs. Reasoning Trade-off: End-to-end SpeechLMs bypass transcription overhead, cutting 90th-percentile Time-to-First-Token (TTFT) from ~800 ms (Whisper v3 + Qwen3 8B) down to ~40 ms (Qwen2.5-Omni 7B). However, because they lack extensive structured agentic pretraining, most SpeechLMs fail to convert this speed advantage into actionable tool calls.
  • Chained Tools Are an Open Bottleneck: In sequential dependent tasks (e.g., querying GPS coordinates and passing them into a ride estimate), all models collapse. Even the best ASR–LLM pipeline tops out at 14.81% PF in English and 7.89% in Hindi.
  • Cultural and Linguistic Fragility: Moving from standard English queries to culturally grounded Indian scenarios drops performance across all models (e.g., KimiAudio 7B drops from 79.8% to 67.8% PF). Non-Hindi Indic languages suffer even larger drops due to sparse multilingual training data.
  • Safety Refusals Evaporate in Non-English Speech: KimiAudio 7B achieves a 51.25% refusal rate on harmful English audio, but this crashes to 1.33% in Hindi and 2.94% on average for other Indic languages.

Key Results

Model Setup English PF (Avg) Hindi PF (Avg) Indic 5-Lang PF (Avg) English Refusal Rate
Qwen2.5-Omni 7B (SpeechLM) 1.69% 0.21% 0.11% 18.75%
AudioFlamingo3 7B (SpeechLM) 15.15% 7.12% 1.99% 7.50%
KimiAudio 7B (SpeechLM) 53.97% 39.43% 25.30% 51.25%
Whisper v3 + Qwen3 8B (ASR-LLM) 52.69% 52.64% 30.34% 52.50%
Whisper v3 + Gemma3 27B (ASR-LLM) 55.08% 50.38% 31.36% 56.25%
Whisper v3 + Llama3 70B (ASR-LLM) 56.26% 54.59% 35.35% 42.50%
  • SpeechLM Divergence: KimiAudio 7B demonstrates that direct speech-to-tool generation is viable (reaching 53.97% English PF), whereas Qwen2.5-Omni and AudioFlamingo3 suffer near-total structural failure (TCS < 2%).
  • ASR Bottleneck: Replacing Whisper v3 transcripts with ground-truth text boosted downstream Indic parameter filling by +24% to +38%. Swapping in an Indic-specialized acoustic model (IndicConformer) recovered 40–55% of this performance gap.

Limitations and Future Work

  • Acoustic Noise: The audio is synthesized in clean acoustic environments; the benchmark does not evaluate background noise or cross-talk.
  • Proprietary Systems Excluded: Due to API costs and closed access, systems like GPT-4o Audio and Gemini 2.5 Pro were not evaluated.
  • Static vs. Interactive: Evaluations are static single-pass or pre-recorded multi-turn conversations rather than real-time, interactive environment simulations.
  • Incomplete Multilingual Coverage: Multi-turn dialogue evaluation was restricted to English and Hindi, leaving five Indic languages without conversational tests.

Summary updated

Abstract

Large-scale Speech Language Models (SpeechLMs) have enabled voice assistants capable of understanding natural spoken queries and performing complex tasks. However, existing speech benchmarks primarily focus on isolated capabilities such as transcription, or question-answering, and do not systematically evaluate agentic scenarios encompassing multilingual and cultural understanding, as well as adversarial robustness. To address this, we introduce VoiceAgentBench, a comprehensive benchmark designed to evaluate SpeechLMs in realistic spoken agentic settings. It comprises over 6,000 synthetic spoken queries, including dialogues grounded in Indian context, covering single-tool invocations, multi-tool workflows, multi-turn interactions, and safety evaluations. The benchmark supports English, Hindi, and 5 other Indian languages, reflecting real-world linguistic and cultural diversity. We simulate speaker variability using a novel sampling algorithm that selects audios for TTS voice conversion based on its speaker embeddings, maximizing acoustic and speaker diversity. Our evaluation measures tool selection accuracy, structural consistency, and the correctness of tool invocations, including adversarial robustness. Our experiments reveal significant gaps in contextual tool orchestration tasks, Indic generalization, and adversarial robustness, exposing critical limitations of current SpeechLMs.
BibTeX
@misc{jain2025voiceagentbench,
  author = {Dhruv Jain and Harshit Shukla and Gautam Rajeev and Ashish Kulkarni and Chandra Khatri and Shubham Agarwal},
  title = {{VoiceAgentBench: Are Voice Assistants ready for agentic tasks?}},
  year = {2025},
  eprint = {2510.07978},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.07978},
}
Figure