TL;DR
Text-based toxicity detectors often flag benign sentences simply because they mention marginalized demographic groups. This paper demonstrates that incorporating raw speech audio at inference time significantly reduces these false positives, particularly for ambiguous or controversial statements. Crucially, the authors find that this bias stems from the classifier itself rather than transcription errors, highlighting the indispensable value of multimodal context.
What Problem Does This Paper Address?
Text toxicity models are notoriously biased against minoritized groups. Because online toxicity frequently targets marginalized communities, machine learning models often conflate the mere mention of identity markers (like "gay", "lesbian", or "Black") with toxic content, leading to high false-positive rates (FPR).
While speech-aware toxicity detection is an emerging area of interest, it hasn't been clear whether operating directly on audio mitigates or worsens these biases. Traditional cascaded systems transcribe speech to text via Automatic Speech Recognition (ASR) before running a text classifier, throwing away rich prosodic information (tone, inflection, pacing). This paper investigates whether direct access to speech data can help classifiers differentiate between actual toxicity and neutral group mentions, and isolates where the bias truly creeps into the pipeline.
Methods
The authors systematically compare four types of toxicity detection systems:
- ETOX: A text-only, wordlist-based baseline.
- DETOXIFY: A text-only neural network trained on Wikipedia and Civil Comments.
- MUTOX-ASR: A multimodal model trained jointly on speech and text, but restricted to using only text embeddings at inference time.
- MUTOX: The full multimodal model, which has access to both raw speech audio and text transcripts during inference.
To evaluate these models fairly, the authors developed a rigorous, multi-stage human annotation pipeline. Three native-level proficient annotators per language labeled samples for toxicity, specific demographic group mentions (gender, sexuality, race, religion, disability, social class), and transcript correctness. Furthermore, they re-evaluated the text-based classifiers using annotator-corrected transcripts to isolate the impact of ASR errors from classifier bias.
Datasets
The study utilizes MUTOX, a large-scale multilingual speech toxicity dataset. The authors focused on the English and Spanish test partitions, creating a new benchmark of 1,954 high-quality group annotations. This represents the first fairness audit dataset specifically designed for evaluating demographic bias in speech toxicity detection.
Key Insights and Ideas
- Acoustic Context is a Shield Against Bias: Raw audio provides crucial prosodic cues that help models realize a group mention is benign. Models with access to audio at inference time show a marked reduction in false positives.
- Inference-Time Speech is Mandatory: Simply training a model on speech data is not enough. If a multimodal model (
MUTOX-ASR) is deprived of audio at inference time, its bias actually worsens. Without acoustic context, the model defaults to using identity terms as shortcuts for toxicity. - Blame the Classifier, Not the ASR: Intuitively, one might assume that bad transcriptions cause false positives. However, feeding perfect, human-corrected transcripts to text classifiers barely changes the false positive rate for group mentions. The bias is baked into the text classification logic, not the transcription pipeline.
Key Results
- Performance Stability: Text-only models see a drop in $F_1$-score when identity groups are mentioned. In contrast,
MUTOX(with speech at inference) maintains or improves precision and recall when groups are present.
- Taming Ambiguity: For ambiguous samples (where human annotators struggled to reach a consensus),
MUTOXachieved a 0% False Positive Rate (FPR). Conversely,MUTOX-ASRshowed a massive spike in FPR when groups were mentioned in ambiguous contexts.
- Demographic Breakdown: When looking at specific identities,
MUTOXconsistently reduced the FPR for mentions of women, Black, and Muslim groups compared to its speech-deprived counterpart,MUTOX-ASR.
- The ASR Correction Experiment: Correcting ASR transcripts marginally improved overall $F_1$-scores for text-based models but had virtually no impact on reducing the FPR for group mentions.
Limitations and Future Work
- Clip Truncation: The MUTOX dataset features short audio clips (2 to 8 seconds). This truncation often removes vital conversational context, leading to an inflated number of "Cannot say" annotations from humans.
- Topic Skew: The authors noted a heavy skew toward liturgical/biblical content in the MUTOX dataset, which complicated gender and toxicity annotations (e.g., quoting violent or historical religious texts).
- Language Scope: The study only covers English and Spanish (high-resource languages). The authors note that for lower-resourced languages—where ASR error rates are significantly higher—transcription errors might play a much larger role in driving pipeline bias than observed here.
- Dialectal Nuances: Annotators were discouraged from inferring speaker demographics, making it difficult to properly evaluate reappropriated language (such as African American English).
Summary updated