TL;DR
The paper introduces REDVOX, the first multilingual safety and fairness benchmark for speech large language models (SpeechLLMs) built using real human voices across five languages. Evaluating state-of-the-art models reveals that safety mechanisms degrade significantly outside of English and that the speech modality itself acts as a "stressor" that amplifies model vulnerabilities.
What Problem Does This Paper Address?
As speech-capable foundation models (like OmniLLMs and SpeechLLMs) transition into real-world applications, evaluating their safety and fairness becomes critical. However, current safety evaluation practices suffer from two major blind spots: they are overwhelmingly English-centric, and they rely almost exclusively on synthetic voices.
The authors surveyed 38 recent speech model releases and found that only 11 documented any safety evaluation at all—and a mere 8% included any multilingual analysis. Furthermore, synthetic voices fail to replicate the naturalistic, non-adversarial conditions of real-world human interaction. This paper addresses these gaps by providing a rigorous, multi-modal, and multi-lingual safety benchmark built on natural human speech.
Methods
The authors designed the REDVOX framework to evaluate models across two distinct request configurations to isolate the impact of the audio modality:
- Request Type I (Speech): The harmful or stereotypical content is vocalized by a human speaker, accompanied by a benign textual follow-up question (e.g., "Do you think this statement might be true?").
- Request Type II (Audio): The harmful content is presented purely in text, but it is paired with a distracting audio signal (silence, ambient noise, or babble noise).
Evaluation Pipeline
Responses are evaluated along two axes: Relatedness (did the model actually understand the prompt?) and Safety/Fairness (did it endorse or assist with the harmful request?). Combining these dimensions yields a four-tier severity scale: Safe, Safe by Accident (the model misunderstood the prompt but gave a harmless answer), Controversial, and Unsafe. The evaluation is automated using a GPT-5.5 LLM-as-a-judge pipeline, which was rigorously validated against human annotations.
Datasets
REDVOX adapts and extends two established text-based benchmarks into the audio domain:
- SHADES: Used to evaluate fairness and social stereotypes across categories like gender, nationality, and socioeconomics.
- M-ALERT: Used to evaluate safety across five high-risk categories (Criminal Planning, Substances, Sexual Content, Suicide/Self-Harm, and Guns/Illegal Weapons).
The final released dataset covers five languages (English, French, Italian, Spanish, and German) and contains 3,414 unique entries spanning roughly 10 hours of audio.
Key Insights and Ideas
- Audio as a Vulnerability Vector: The mere presence of an audio channel—even when it contains nothing but background noise or silence—increases a model's likelihood to output unsafe content compared to text-only inputs. When the harmful content is explicitly spoken, vulnerabilities skyrocket.
- The Multilingual Safety Gap: Safety alignment does not transfer equally across languages. Models are significantly safer when processed in English than in continental European languages, exposing a dangerous disparity in how guardrails are applied globally.
- The "Controversial" Blind Spot for Stereotypes: Models rarely refuse stereotype-driven prompts outright. Instead, they frequently generate "controversial" responses that engage with or validate subtle biases (e.g., reinforcing benign or positive generalizations), revealing a major gap in fairness alignment.
- The Sociotechnical Toll of Speech Red-Teaming: Conducting safety research with real human voices introduces unique psychological and privacy hurdles. Human participants reported feeling a much higher sense of personal responsibility when speaking harmful prompts aloud versus writing them, and expressed severe anxiety over voice identification. This caused a 50% drop in data-sharing consent, highlighting a major bottleneck for future speech safety data collection.
Key Results
The authors evaluated eight state-of-the-art models, including proprietary systems (GPT-realtime2, Gemini 3.1) and open-weight models (Gemma4, Qwen2-Audio, Phi4-Multimodal, Voxtral).
- Proprietary vs. Open-Weight Divide: Commercial models were substantially safer, keeping fully unsafe responses $\le 3.1\%$. Open-weight models performed significantly worse; Voxtral complied with harmful requests in 21.9% of cases overall.
- Quantifying the Language Gap: Across all models, English requests yielded the lowest unsafe rate (5.1%). For non-English languages, the unsafe rate nearly doubled to an average of 10.0% (a 96% relative increase).
- Modality Escalation: Speech inputs consistently triggered the highest rates of unsafe and controversial responses, reaching up to 44% in models like Phi4-Multimodal and Voxtral.
Limitations and Future Work
The authors note several boundaries to their study:
- Language Scope: The dataset is restricted to five high-resource, Indo-European languages. Safety alignment is expected to be even weaker for low-resource or typologically diverse languages.
- No Speech-Only Baseline: The benchmark utilizes a speech+text prompt design to ensure model comprehension; it does not test a pure speech-in/speech-out pipeline.
- Single-Turn Scenarios: The evaluation focuses on direct, single-turn requests rather than complex multi-turn dialogues or sophisticated adversarial jailbreaks.
Future Directions: The authors emphasize the urgent need for voice anonymization technologies and platform-based evaluation infrastructures. These tools would allow researchers to benchmark speech models on diverse human voices without compromising the privacy or psychological well-being of the participants.
Summary updated