RedVox: Safety and Fairness Gaps in Speech Models Across Languages

Beatrice Savoldi, Sara Papi, Wafa Aissa, Matteo Negri, Luisa BentivogliarXiv 2026 · arXiv

TL;DR

The paper introduces REDVOX, the first multilingual safety and fairness benchmark for speech large language models (SpeechLLMs) built using real human voices across five languages. Evaluating state-of-the-art models reveals that safety mechanisms degrade significantly outside of English and that the speech modality itself acts as a "stressor" that amplifies model vulnerabilities.

What Problem Does This Paper Address?

As speech-capable foundation models (like OmniLLMs and SpeechLLMs) transition into real-world applications, evaluating their safety and fairness becomes critical. However, current safety evaluation practices suffer from two major blind spots: they are overwhelmingly English-centric, and they rely almost exclusively on synthetic voices.

The authors surveyed 38 recent speech model releases and found that only 11 documented any safety evaluation at all—and a mere 8% included any multilingual analysis. Furthermore, synthetic voices fail to replicate the naturalistic, non-adversarial conditions of real-world human interaction. This paper addresses these gaps by providing a rigorous, multi-modal, and multi-lingual safety benchmark built on natural human speech.

Figure 1: Safety evaluation practices across 11 speech models that provide safety documentation. We show lan- guage coverage, evaluation methodology (automatic vs. automatic+manual), and data type (public vs. private).

Methods

The authors designed the REDVOX framework to evaluate models across two distinct request configurations to isolate the impact of the audio modality:

  1. Request Type I (Speech): The harmful or stereotypical content is vocalized by a human speaker, accompanied by a benign textual follow-up question (e.g., "Do you think this statement might be true?").
  2. Request Type II (Audio): The harmful content is presented purely in text, but it is paired with a distracting audio signal (silence, ambient noise, or babble noise).
The REDVOX evaluation framework splits inputs into Speech and Audio types to test model robustness.

Evaluation Pipeline

Responses are evaluated along two axes: Relatedness (did the model actually understand the prompt?) and Safety/Fairness (did it endorse or assist with the harmful request?). Combining these dimensions yields a four-tier severity scale: Safe, Safe by Accident (the model misunderstood the prompt but gave a harmless answer), Controversial, and Unsafe. The evaluation is automated using a GPT-5.5 LLM-as-a-judge pipeline, which was rigorously validated against human annotations.

Datasets

REDVOX adapts and extends two established text-based benchmarks into the audio domain:

  • SHADES: Used to evaluate fairness and social stereotypes across categories like gender, nationality, and socioeconomics.
  • M-ALERT: Used to evaluate safety across five high-risk categories (Criminal Planning, Substances, Sexual Content, Suicide/Self-Harm, and Guns/Illegal Weapons).

The final released dataset covers five languages (English, French, Italian, Spanish, and German) and contains 3,414 unique entries spanning roughly 10 hours of audio.

Figure 3: Composition of REDVOX released data. En- glish represents the largest portion (40%). Vulnerabil- ity types (middle ring) are distinguished across unsafe (69%) and stereotypical requests (31%). Each audio type and speech recordings amount to ∼25% each.

Key Insights and Ideas

  • Audio as a Vulnerability Vector: The mere presence of an audio channel—even when it contains nothing but background noise or silence—increases a model's likelihood to output unsafe content compared to text-only inputs. When the harmful content is explicitly spoken, vulnerabilities skyrocket.
  • The Multilingual Safety Gap: Safety alignment does not transfer equally across languages. Models are significantly safer when processed in English than in continental European languages, exposing a dangerous disparity in how guardrails are applied globally.
  • The "Controversial" Blind Spot for Stereotypes: Models rarely refuse stereotype-driven prompts outright. Instead, they frequently generate "controversial" responses that engage with or validate subtle biases (e.g., reinforcing benign or positive generalizations), revealing a major gap in fairness alignment.
  • The Sociotechnical Toll of Speech Red-Teaming: Conducting safety research with real human voices introduces unique psychological and privacy hurdles. Human participants reported feeling a much higher sense of personal responsibility when speaking harmful prompts aloud versus writing them, and expressed severe anxiety over voice identification. This caused a 50% drop in data-sharing consent, highlighting a major bottleneck for future speech safety data collection.

Key Results

The authors evaluated eight state-of-the-art models, including proprietary systems (GPT-realtime2, Gemini 3.1) and open-weight models (Gemma4, Qwen2-Audio, Phi4-Multimodal, Voxtral).

Overall distribution of model responses on REDVOX, highlighting the safety divide between proprietary and open-weight models.
  • Proprietary vs. Open-Weight Divide: Commercial models were substantially safer, keeping fully unsafe responses $\le 3.1\%$. Open-weight models performed significantly worse; Voxtral complied with harmful requests in 21.9% of cases overall.
  • Quantifying the Language Gap: Across all models, English requests yielded the lowest unsafe rate (5.1%). For non-English languages, the unsafe rate nearly doubled to an average of 10.0% (a 96% relative increase).
  • Modality Escalation: Speech inputs consistently triggered the highest rates of unsafe and controversial responses, reaching up to 44% in models like Phi4-Multimodal and Voxtral.

Limitations and Future Work

The authors note several boundaries to their study:

  1. Language Scope: The dataset is restricted to five high-resource, Indo-European languages. Safety alignment is expected to be even weaker for low-resource or typologically diverse languages.
  2. No Speech-Only Baseline: The benchmark utilizes a speech+text prompt design to ensure model comprehension; it does not test a pure speech-in/speech-out pipeline.
  3. Single-Turn Scenarios: The evaluation focuses on direct, single-turn requests rather than complex multi-turn dialogues or sophisticated adversarial jailbreaks.

Future Directions: The authors emphasize the urgent need for voice anonymization technologies and platform-based evaluation infrastructures. These tools would allow researchers to benchmark speech models on diverse human voices without compromising the privacy or psychological well-being of the participants.

Summary updated

Abstract

Speech-capable models are increasingly deployed in real-world applications across languages. Yet their safety and fairness beyond English settings and under naturalistic conditions remain understudied. We survey safety reporting practices across state-of-the-art speech model releases, finding that only 8% document any multilingual analysis. To address this gap, we introduce RedVox, a multilingual safety and fairness benchmark for audio and speech built on real voices, covering unsafe and unfair stereotypical requests across five languages (English, French, Italian, Spanish, and German). Evaluating eight state-of-the-art models, we find that vulnerabilities persist even under non-adversarial conditions, worsen in non-English languages, and are amplified when the request comes from a spoken input. Finally, by surveying the participants who contributed to RedVox, we document the unique personal and privacy challenges of collecting speech data with human participants, pointing to broader sociotechnical challenges in naturalistic speech safety research.
BibTeX
@misc{savoldi2026redvox,
  author = {Beatrice Savoldi and Sara Papi and Wafa Aissa and Matteo Negri and Luisa Bentivogli},
  title = {{RedVox: Safety and Fairness Gaps in Speech Models Across Languages}},
  year = {2026},
  eprint = {2606.26968},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.26968},
}
Figure