Aegis: Towards Governance, Integrity, and Security of AI Voice Agents

Xiang Li, Pin-Yu Chen, Wenqi WeiSubmission to ICLR 2026 · Rejected

TL;DR

Aegis is the first systematic red-teaming framework designed to evaluate the security of Audio Large Language Model (ALLM) voice agents in real-world deployment scenarios. By simulating multi-turn interactions in high-stakes domains like banking and IT support, the researchers discovered that while strict data access controls can stop privacy leaks, they fail to prevent behavioral threats like privilege escalation and resource abuse.

What Problem Does This Paper Address?

As Audio Large Language Models (ALLMs) move from research labs to production, they are being integrated into voice agents for banking, logistics, and IT support. Unlike traditional text-based LLMs, these agents operate in continuous, real-time, multi-turn environments where they often have access to sensitive backend databases and functional tools.

Current security benchmarks primarily focus on "model-level" robustness—essentially trying to get a model to say something offensive or leak a secret in a single turn. However, a real-world attacker doesn't just send one prompt; they engage in a conversation, using social engineering, linguistic cues, and persistence to bypass safeguards. There is a critical gap in understanding how these agents behave when integrated into a full service pipeline. The authors introduce Aegis to bridge this gap by shifting the focus from model-level safety to agent-level governance.

Methods

The authors developed a red-teaming framework that simulates end-to-end deployment pipelines across three representative domains: Bank Call Centers, IT Support Desks, and Logistics Dispatch.

Key components of the Aegis framework include:

  • Adversarial Scenarios: The framework maps five critical risks based on the MITRE ATT&CK framework: Authentication Bypass, Privacy Leakage, Resource Abuse, Privilege Escalation, and Data Poisoning.
  • Automated Attacker Agent: The researchers used GPT-4o to act as an adversary. This attacker is given a high-level objective and one of five personas (e.g., a "disgruntled employee" or a "friendly manipulator") to induce diverse tactical styles.
  • Database Access Paradigms: They tested two ways agents interact with data: Direct Read Access (where the agent sees raw records) and Query-based Access (where the agent uses an intermediary layer).
  • Multi-turn Evaluation: Unlike static benchmarks, Aegis allows for up to 10 turns of dialogue, letting the attacker refine their tactics based on the agent's responses.

Datasets

Rather than using a single static dataset, the authors constructed backend databases for each of the three domains:

  • Banking: Customer profiles, account balances, and transaction histories.
  • IT Support: Employee credentials, access rights, and asset inventories.
  • Logistics: Shipment records, delivery addresses, and scheduling details.

The framework was used to evaluate seven state-of-the-art ALLM backbones, including the GPT-4o family, Gemini 1.5/2.5 series, and the open-weight Qwen2 Audio/2.5-omni models.

Key Insights and Ideas

  • Access Control is Not a Silver Bullet: The most significant finding is that restricting an agent to "query-based" access (preventing it from seeing raw data) successfully eliminated authentication bypass and privacy leaks. However, it did nothing to stop behavioral threats like privilege escalation or resource abuse. This suggests that vulnerabilities driven by the model's "tendency to comply" cannot be fixed by data policies alone.
  • The Open-Weight Vulnerability Gap: Open-weight models (like Qwen) consistently showed higher susceptibility to adversarial manipulation compared to closed-source counterparts (like Gemini or GPT-4o). This highlights a lag in safety alignment for open-source audio models.
  • Resource Abuse is the Hardest to Defend: Across all models, "Resource Abusing" (getting the agent to perform irrelevant tasks or waste computation) had the highest success rates, often exceeding 60-70%.
  • Subtle Cues Matter: While not dominant, the researchers found that gender cues and specific personas (like an "urgent" customer) could subtly increase an agent's likelihood of bypassing its own security protocols.

Key Results

  • Data Access Impact: In the "Direct Read" setting, authentication bypass success rates were as high as 20.8%. Switching to "Query-based" access dropped this to 0% across all models.
  • Model Performance: Gemini-2.5 Pro demonstrated the strongest robustness, maintaining the lowest attack success rates across all scenarios. In contrast, Qwen2 Audio was the most vulnerable, with a 14.8% success rate for privilege escalation even under strict query-based conditions.
  • Persistence of Behavioral Risks: Even with the best defenses, privilege escalation (up to 14.8%) and data poisoning (up to 15.2%) remained non-negligible threats, proving that attackers can still manipulate system behavior without needing raw data access.

Limitations and Future Work

The authors acknowledge that while Aegis is a major step forward, it currently relies on an LLM-based evaluator to judge attack success, which may have its own biases.

For future work, they suggest:

  1. Human-in-the-loop: Incorporating human oversight for high-impact actions (like fund transfers).
  2. Dynamic Policy Adaptation: Developing systems that can change their security posture based on the context and history of a conversation.
  3. Behavioral Anomaly Detection: Integrating real-time monitoring to flag and throttle users who exhibit "attacker-like" conversational patterns.

Summary updated

Abstract

With the rapid advancement and adoption of Audio Large Language Models (ALLMs), voice agents are now being deployed in high-stakes domains such as banking, customer service, and IT support. However, their vulnerabilities to adversarial misuse still remain unexplored. While prior work has examined aspects of trustworthiness in ALLMs, such as harmful content generation and hallucination, systematic security evaluations of voice agents are still lacking. To address this gap, we propose Aegis, a red-teaming framework for the governance, integrity, and security of voice agents. Aegis models the realistic deployment pipeline of voice agents and designs structured adversarial scenarios of critical risks, including privacy leakage, privilege escalation, resource abuse, etc. We evaluate the framework through case studies in banking call centers, IT Support, and logistics. Our evaluation reveals several important findings. First, restricting agents to query-based database access eliminates authentication bypass and privacy leakage attacks. However, behavioral threats such as privilege escalation, instruction poisoning, and resource abuse persist even under stricter access controls, indicating that compliance-driven vulnerabilities cannot be mitigated by data access policies alone. Moreover, open-weight models show consistently higher susceptibility to adversarial manipulation compared to closed-source ones. In addition, we also found that attacker personas and gender cues can influence outcomes but are not dominant factors when strong operational policies are enforced. These insights underscore the necessity of layered defense strategies-combining access control, policy enforcement, and behavioral monitoring- to secure next-generation ALLM-powered voice agents.
BibTeX
@misc{li2026aegis,
  author = {Xiang Li and Pin-Yu Chen and Wenqi Wei},
  title = {{Aegis: Towards Governance, Integrity, and Security of AI Voice Agents}},
  year = {2026},
  eprint = {2602.07379},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.07379},
}
Figure