VoiceBBQ: Investigating Effect of Content and Acoustics in Social Bias of Spoken Language Model

Junhyuk Choi, Ro-hoon Oh, Jihwan Seol, Bugeun KimEMNLP 2025 · OpenReview

TL;DR

This paper introduces VoiceBBQ, a novel benchmark for evaluating social bias in spoken language models (SLMs) by extending the text-based BBQ dataset to include acoustic variations. They found that SLM architectures significantly influence how content and acoustic cues contribute to bias, with LLaMA-Omni being more sensitive to acoustic variations than Qwen2-Audio.

What Problem Does This Paper Address?

As AI systems become more integrated into our lives, understanding and mitigating social bias is crucial. While research has explored bias in text-based models, spoken language models (SLMs) present a unique challenge. Social bias in SLMs can arise not only from the content of what is said but also from the acoustic characteristics of the speaker (e.g., gender, accent). Existing research has largely focused on content bias, leaving a gap in understanding how acoustic features influence SLM bias. This paper aims to bridge this gap by systematically disentangling the effects of content and acoustics on social bias in SLMs.

Methods

The core of this work is the creation of VoiceBBQ, a spoken version of the Bias Benchmark for Question answering (BBQ). The authors synthesized speech for the context portions of the original BBQ dataset, creating 58,492 spoken examples. They controlled for two key acoustic factors: gender (male/female) and accent (American/British), generating 16 unique voice conditions for each context.

To evaluate bias, they employed a two-pronged approach:

  • Content-Aspect Analysis: They analyzed how SLMs inherited biases from their underlying large language models (LLMs). This involved comparing bias scores of SLMs with their respective backbone LLMs and calculating Pearson correlations. To isolate content bias, they averaged results across all acoustic conditions.
  • Acoustic-Aspect Analysis: They investigated whether SLM predictions varied across different acoustic conditions (gender and accent) while keeping the content identical. This was done by comparing model responses under these controlled acoustic variations and using McNemar's test to assess statistical significance.

Datasets

  • BBQ (Bias Benchmark for Question answering): The original text-based dataset used as the foundation for VoiceBBQ.
  • VoiceBBQ: A newly introduced benchmark created by synthesizing speech for the context portions of the BBQ dataset. It comprises 58,492 spoken examples with controlled variations in gender and accent.

Key Insights and Ideas

  • Dual Sources of Bias: The paper clearly demonstrates that social bias in SLMs is a complex interplay of both what is said (content) and how it is said (acoustics). This highlights the need for evaluation methods that consider both aspects.
  • Architectural Influence: The study reveals that the architectural choices in SLMs play a significant role in how they process and are affected by acoustic cues. This suggests that bias mitigation strategies might need to be architecture-specific.
  • Content Inheritance Varies: SLMs don't uniformly inherit biases from their parent LLMs. The degree of inheritance depends on the SLM's architecture and potentially its training data, as seen in the differing correlations between LLaMA-Omni and Qwen2-Audio with their respective backbones.
  • Acoustic Sensitivity is Not Uniform: While some SLMs are highly sensitive to acoustic variations (like LLaMA-Omni), others effectively dampen these cues (like Qwen2-Audio), leading to different bias profiles.

Key Results

  • Content Bias:
    • Qwen2-Audio showed a strong correlation (r ≈ 0.84) with its backbone LLM (Qwen1) in terms of content bias, indicating significant inheritance.
  • LLaMA-Omni showed a weaker correlation (r = 0.62 to 0.30) with its backbone LLM (LLaMA 3.1), suggesting its biases were reshaped, possibly by its training data, leading to lower overall bias scores in many categories compared to Qwen2-Audio.
  • Acoustic Bias:
    • LLaMA-Omni exhibited significant variations in responses across gender and accent conditions, particularly for categories like Gender Identity, Race/Ethnicity, and SES. This indicates it amplifies acoustic bias.
  • Qwen2-Audio showed near-zero bias score differences across gender and accent conditions, suggesting it effectively mitigates acoustic bias. This is attributed to its architecture, which integrates and trains the audio encoder end-to-end, potentially smoothing out speaker-specific information.

Limitations and Future Work

  • Limited Model Scope: The study evaluated only two SLM architectures (LLaMA-Omni and Qwen2-Audio). Broader analysis across more models is needed to generalize findings.
  • No Mitigation Strategies: The paper focuses on diagnosing bias rather than proposing methods to mitigate it. Future work will involve designing and evaluating SLM architectures that actively reduce identified biases.
  • Sociocultural Mechanisms Unexplored: While statistical associations between acoustic features and bias are identified, the underlying sociocultural reasons for these patterns (e.g., why male voices might trigger stronger gender identity biases) remain an open question for further exploration.

Summary updated

Abstract

We introduce VoiceBBQ, a spoken extension of the BBQ (Bias Benchmark for Question answering) - a dataset that measures social bias by presenting ambiguous or disambiguated contexts followed by questions that may elicit stereotypical responses. Due to the nature of speech modality, social bias in Spoken Language Models (SLMs) can emerge from two distinct sources: 1) content aspect and 2) acoustic aspect. The dataset converts every BBQ context into controlled voice conditions, enabling per-axis accuracy, bias, and consistency scores that remain comparable to the original text benchmark. Using VoiceBBQ, we evaluate two SLMs—LLaMA-Omni and Qwen2-Audio—and observe architectural contrasts: LLaMA-Omni retains strong acoustic sensitivity, amplifying gender and accent bias, whereas Qwen2-Audio substantially dampens these cues while preserving content fidelity. VoiceBBQ thus provides a compact, drop-in testbed for jointly diagnosing content and acoustic bias across spoken language models.
BibTeX
@inproceedings{choi2025voicebbq,
  author = {Junhyuk Choi and Ro-hoon Oh and Jihwan Seol and Bugeun Kim},
  title = {{VoiceBBQ: Investigating Effect of Content and Acoustics in Social Bias of Spoken Language Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=1QrXeKoSxq},
}
Figure