Acoustic and Paralinguistic Bias in Speech Models

36 concepts16 papers1 proposal

Traditional natural language processing studies social bias by analyzing text tokens. In text, a bias trigger is explicit: a specific pronoun, a gendered noun, or a culturally loaded name. However, as language models evolve to directly "listen" and "speak" via integrated speech encoders, a new dimension of algorithmic bias emerges.

When a model processes raw audio, it handles a dual stream of information: the semantic content (the words being spoken) and the paralinguistic or acoustic features (how those words are spoken, including pitch, tone, resonance, and accent). Recent research reveals that speech-integrated models inadvertently use these non-semantic acoustic features as proxies for demographic attributes, triggering social stereotypes and systematic performance disparities that text-only evaluations entirely miss.

How Acoustic Features Trigger Bias

Why do speech models exhibit bias even when the spoken text is completely neutral? The root cause lies in how speech encoders map continuous audio signals into latent representations.

When an audio waveform is processed by a speech encoder (such as Whisper), the network extracts acoustic patterns to understand the text. However, features like fundamental frequency (pitch) and formant transitions are highly correlated with speaker demographics like gender, age, and regional origin. As Beatrice Savoldi et al. (2022) demonstrate in speech translation tasks, models frequently perform "gender profiling." Instead of relying on contextual linguistic cues to determine grammatical agreement, they rely on acoustic features like pitch and tone as shortcuts.

This reliance creates a phenomenon known as paradoxical gender bias. As documented by Junhyuk Choi et al. (2025), speech-aware language models often respond with male stereotypes when presented with gender-neutral questions spoken by a male voice, yet they completely fail to differentiate or utilize gender information when it is contextually relevant to the task. By systematically isolating the components of these models, the authors traced this failure directly back to the speech encoder rather than the downstream language model. The encoder passes acoustic demographic cues in a way that activates latent stereotypes in the language model, but fails to structure it as actionable contextual information.

[Raw Audio Input] ──> [Speech Encoder] ───> [Latent Embeddings] ───> [Language Model] ───> [Biased Output]
  (Words + Pitch/Tone)    (Extracts Acoustic     (Demographic Proxies     (Activates Latent
                           Features)              Leaked Intentionally)    Stereotypes)

Internal Representations vs. Downstream Propagation

A common intuition might suggest that the solution to acoustic bias is to make speech encoders entirely blind to speaker demographics. However, internal probing of these architectures reveals a more complex reality: hiding demographic information can actually worsen performance gaps.

Beatrice Savoldi et al. (2024) investigated state-of-the-art multilingual speech recognition models, including Whisper and SeamlessM4T, which exhibit significant performance gaps across diverse languages. By probing the models' internal hidden states, the authors discovered that the more clearly a model represents gender in its latent space, the smaller the performance gap becomes for female speakers. When a model explicitly disentangles speaker characteristics from the linguistic signal, it avoids confusing acoustic variance with phonetic noise, leading to more equitable transcription accuracy.

When these acoustic representations are poorly managed, the embedded biases propagate downstream into sensitive applications. Isaac Slaughter et al. (2023) developed the Speech Embedding Association Test (SpEAT) to quantify these hidden associations. They demonstrated that human-like social biases regarding race, age, and disability are deeply embedded within popular pre-trained speech checkpoints. Crucially, these foundational biases propagate directly to downstream tasks like Speech Emotion Recognition (SER), causing systems to systematically misinterpret the emotional states of specific demographic groups. This aligns with earlier findings by Cristina Gorrostieta et al. (2019), who noted that SER models consistently underperform on female voices in naturalistic conversational data, requiring explicit acoustic de-biasing layers to achieve parity.

Disentangling Content from Acoustics

Evaluating acoustic bias is uniquely challenging because text and sound are fundamentally intertwined. If a speech model generates a biased response, it is difficult to determine whether the bias was triggered by the semantic meaning of the words or the acoustic profile of the speaker's voice.

To isolate these variables, researchers have developed frameworks that decouple text from audio:

  • Acoustic Control: Junhyuk Choi et al. (2025) introduced VoiceBBQ, a benchmark that takes identical textual content and renders it across varied speaker genders and accents. By applying statistical tests to the model's outputs across these identical text-but-different-audio pairs, the framework isolates pure acoustic bias. Their evaluations revealed that model architecture heavily dictates vulnerability; for instance, LLaMA-Omni proved significantly more sensitive to acoustic variations than Qwen2-Audio.
  • Auditory Task Adaptation: Yi-Cheng Lin et al. (2024) developed a framework that translates classic text-based bias benchmarks into auditory reasoning tasks, such as spoken coreference resolution. This allows researchers to measure exactly how gender stereotypes are activated by spoken context across different architectural configurations.

Compounding the evaluation challenge is the issue of linguistic bias in automated judges. As Shu-wen Yang et al. (2026) point out, when researchers use an LLM to evaluate whether a speech-to-speech model responded with appropriate paralinguistic empathy or tone, the automated judge often reads the text transcript of the response rather than listening to the actual audio features. This creates an illusion of alignment, masking whether the model is truly processing vocal cues or simply guessing intent from textual context.

The Frontier of Speech LLM Trustworthiness

As speech models transition from passive transcribers to interactive agents, acoustic biases manifest in generative behaviors. For example, in speech-to-speech models that automatically select an output voice profile without explicit user instructions, the model must make an implicit choice. Dariia Puhach et al. (2025) demonstrate that analyzing automatic speaker assignment serves as a direct diagnostic tool for uncovering a generative model's internal gender inclinations and demographic associations.

To address these multifaceted risks, comprehensive safety frameworks like AudioTrust (Kai Li et al. (2026)) and evaluation benchmarks like Spoken Stereoset (Hung-yi Lee et al. (2024)) have emerged. These frameworks systematically test models across dimensions of fairness, robustness, and safety, repeatedly confirming that non-semantic features like regional accents and emotional delivery styles trigger stronger algorithmic biases and vulnerabilities than traditional text-based attributes.

While the community has established robust methods to detect and quantify how paralinguistic features trigger harmful stereotypes, developing training and fine-tuning strategies to effectively de-bias speech encoders without degrading their acoustic sensitivity remains an open and active area of research.

Key Foundational Work

Language ID Prediction from Speech Using Self-Attentive Pooling (2021) — contributed 2 concept elements to this direction.
Gender De-Biasing in Speech Emotion Recognition (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_paradoxical_gender_bias_cf2ded82,
  title = {paradoxical gender bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/paradoxical-gender-bias/rdr_cf2ded82e021438983c3074c36af87c1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_paradoxical_gender_bias_cf2ded82_v1,
  title = {paradoxical gender bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/paradoxical-gender-bias/rdr_cf2ded82e021438983c3074c36af87c1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake