Abstract
AURA is an open-source, speech-to-speech assistant designed to bridge the gap between text-based agentic reasoning and voice-driven interfaces. By integrating multi-turn reasoning with real-world tool use—including email, calendar management, and web search—AURA achieves a 90% success rate on complex tasks. Utilizing a modular cascaded architecture powered by LLaMA-3.3-70B, the system demonstrates performance competitive with proprietary models like GPT-4o, establishing a new baseline for open-source voice agents.
Introduction
Traditional voice assistants typically function as simple "command-and-control" systems or as experimental end-to-end (E2E) speech models that struggle with multi-step reasoning. While text-based Large Language Models (LLMs) have excelled at tool use, bringing this "agentic" capability to a seamless voice interface has remained a significant challenge. AURA addresses this by providing a framework capable of handling compound user requests, such as coordinating schedules and communicating details via email, through a robust reasoning-action loop.
Methods and Architecture
AURA (Agent for Understanding, Reasoning, and Automated Tool Use) employs a cascaded architecture, which the authors identify as superior to current E2E models for complex task execution. The system is organized into four primary modules:
- User Interface (UI): A Gradio-based interface that manages voice input and provides synchronized audio responses and live transcripts.
- Speech Processing: Utilizes Whisper-v3-large or OWSM for Automatic Speech Recognition (ASR) and ESPnet-TTS for Text-to-Speech (TTS).
- Dialog Processing Unit (DPU): The system's core, following the ReAct (Reason + Act) paradigm to interleave reasoning steps with specific tool actions.
- LLM Server: Powered by LLaMA-3.3-70B-Instruct hosted via vLLM, which manages both agentic reasoning and Dialog State Tracking (DST).
The system supports five core action types: Chat, Calendar, Web Search, Contact lookup, and Email.
Evaluation and Results
The researchers evaluated AURA across several benchmarks, including VoiceBench (OpenBookQA and AlpacaEval) for general knowledge, SpokenWOZ for Dialog State Tracking, and a custom set of 30 human-evaluated compound tasks.
Performance Benchmarks
AURA demonstrated near-proprietary performance levels. On the VoiceBench OpenBookQA task, AURA achieved 92.75% accuracy, nearly matching the 92.97% of the closed-source GPT-4o pipeline. In human evaluations, the system maintained a 90% overall success rate, successfully completing 83.33% of tasks classified as "Hard."
Key Insights
- Agentic Reasoning: Prompting the agent to perform a web search before answering improved accuracy on OpenBookQA from 87.69% to 89.23%, highlighting the value of "thinking" before acting to reduce hallucinations.
- Cascaded vs. E2E: AURA's cascaded approach significantly outperformed current E2E speech-to-speech models like Moshi or Mini-Omni on practical benchmarks, suggesting that high-performing text LLMs are currently more adaptable for complex tool use.
- Zero-Shot DST: AURA achieved state-of-the-art results on the SpokenWOZ benchmark with a 28.76% Joint Goal Accuracy (JGA) without specific fine-tuning.
Limitations and Future Work
While AURA sets a high bar for open-source agents, it faces inherent latency challenges due to the turn-taking nature of cascaded pipelines. Future work will focus on expanding the toolset and implementing more robust security frameworks for autonomous actions. The modular design ensures that AURA can easily integrate more complex APIs as they become available.
Summary updated