AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks

Leander Melroy Maben, Gayathri Ganesh Lakshmy, Srijith Radhakrishnan, Siddhant Arora, Shinji WatanabearXiv 2025 · DOI · arXiv

Abstract

AURA is an open-source, speech-to-speech assistant designed to bridge the gap between text-based agentic reasoning and voice-driven interfaces. By integrating multi-turn reasoning with real-world tool use—including email, calendar management, and web search—AURA achieves a 90% success rate on complex tasks. Utilizing a modular cascaded architecture powered by LLaMA-3.3-70B, the system demonstrates performance competitive with proprietary models like GPT-4o, establishing a new baseline for open-source voice agents.

Introduction

Traditional voice assistants typically function as simple "command-and-control" systems or as experimental end-to-end (E2E) speech models that struggle with multi-step reasoning. While text-based Large Language Models (LLMs) have excelled at tool use, bringing this "agentic" capability to a seamless voice interface has remained a significant challenge. AURA addresses this by providing a framework capable of handling compound user requests, such as coordinating schedules and communicating details via email, through a robust reasoning-action loop.

Methods and Architecture

AURA (Agent for Understanding, Reasoning, and Automated Tool Use) employs a cascaded architecture, which the authors identify as superior to current E2E models for complex task execution. The system is organized into four primary modules:

  1. User Interface (UI): A Gradio-based interface that manages voice input and provides synchronized audio responses and live transcripts.
  2. Speech Processing: Utilizes Whisper-v3-large or OWSM for Automatic Speech Recognition (ASR) and ESPnet-TTS for Text-to-Speech (TTS).
  3. Dialog Processing Unit (DPU): The system's core, following the ReAct (Reason + Act) paradigm to interleave reasoning steps with specific tool actions.
  4. LLM Server: Powered by LLaMA-3.3-70B-Instruct hosted via vLLM, which manages both agentic reasoning and Dialog State Tracking (DST).
The modular cascaded architecture of AURA, illustrating the flow between speech modules, the central controller, and external tool APIs.

The system supports five core action types: Chat, Calendar, Web Search, Contact lookup, and Email.

Evaluation and Results

The researchers evaluated AURA across several benchmarks, including VoiceBench (OpenBookQA and AlpacaEval) for general knowledge, SpokenWOZ for Dialog State Tracking, and a custom set of 30 human-evaluated compound tasks.

Performance Benchmarks

AURA demonstrated near-proprietary performance levels. On the VoiceBench OpenBookQA task, AURA achieved 92.75% accuracy, nearly matching the 92.97% of the closed-source GPT-4o pipeline. In human evaluations, the system maintained a 90% overall success rate, successfully completing 83.33% of tasks classified as "Hard."

Performance comparison of AURA against proprietary and open-source models on VoiceBench benchmarks.

Key Insights

  • Agentic Reasoning: Prompting the agent to perform a web search before answering improved accuracy on OpenBookQA from 87.69% to 89.23%, highlighting the value of "thinking" before acting to reduce hallucinations.
  • Cascaded vs. E2E: AURA's cascaded approach significantly outperformed current E2E speech-to-speech models like Moshi or Mini-Omni on practical benchmarks, suggesting that high-performing text LLMs are currently more adaptable for complex tool use.
  • Zero-Shot DST: AURA achieved state-of-the-art results on the SpokenWOZ benchmark with a 28.76% Joint Goal Accuracy (JGA) without specific fine-tuning.

Limitations and Future Work

While AURA sets a high bar for open-source agents, it faces inherent latency challenges due to the turn-taking nature of cascaded pipelines. Future work will focus on expanding the toolset and implementing more robust security frameworks for autonomous actions. The modular design ensures that AURA can easily integrate more complex APIs as they become available.

Summary updated

Abstract

Despite advances in language and speech technologies, no open-source system enables full speech-to-speech, multi-turn dialogue with integrated tool use and agentic reasoning. We introduce AURA (Agent for Understanding, Reasoning, and Automated Tool Use), the first open-source, speech-native assistant capable of completing complex, goal-driven tasks through dynamic tool invocation and multi-turn conversation. AURA combines open-weight ASR, TTS, and LLMs in a cascaded pipeline and supports tools such as calendar booking, contact lookup, web search, and email. Its modular design allows easy integration of new tools using natural language prompts and action classes. On VoiceBench, AURA scores 92.75% on OpenBookQA-outperforming all open-weight systems and nearing GPT-4o-and 4.39 on AlpacaEval, competitive with other open-weight systems. Human evaluation shows 90% task success on complex, multi-turn speech tasks.
BibTeX
@misc{maben2025aura,
  author = {Leander Melroy Maben and Gayathri Ganesh Lakshmy and Srijith Radhakrishnan and Siddhant Arora and Shinji Watanabe},
  title = {{AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks}},
  year = {2025},
  eprint = {2506.23049},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.23049},
}
Figure