TL;DR
This paper investigates whether using speech data can reduce biases in toxicity detection systems, which often disproportionately flag content mentioning demographic groups as toxic. By creating new annotations for a speech toxicity dataset and comparing speech-based models to text-based ones, they found that speech data at inference time significantly reduces false positives for group mentions, especially in ambiguous cases.
What Problem Does This Paper Address?
Toxicity detection systems, widely used for content moderation, are known to exhibit significant biases. Specifically, they tend to flag content mentioning marginalized groups (like racial or ethnic minorities, LGBTQ+ individuals, etc.) as toxic, even when the content is benign. This often stems from biases in the training data, where these groups are disproportionately the subject of actual toxicity. Traditional approaches to speech toxicity detection involve converting speech to text first (ASR), which can lose valuable prosodic and contextual information. This paper asks: can directly processing speech, rather than just its text transcription, help mitigate the biases seen in text-based toxicity detection?
Methods
The researchers conducted a systematic comparison of four toxicity detection systems:
- ETOX: A text-only wordlist-based classifier.
- DETOXIFY: A text-only neural network classifier.
- MUTOX-ASR: A neural network trained on speech and text, but only using text embeddings at inference.
- MUTOX: A neural network trained on speech and text, and crucially, using both speech audio and text embeddings at inference.
They introduced a new set of high-quality annotations for the MUTOX dataset, focusing on toxicity and group mentions, and also corrected ASR transcripts. This allowed them to evaluate models on their performance and bias against group mentions, particularly on ambiguous samples. They also investigated the impact of ASR transcription errors by re-evaluating models with corrected transcripts.
Datasets
- MUTOX dataset: The primary dataset used for evaluation. The researchers created new group annotations for the English and Spanish test partitions of this multilingual speech toxicity dataset.
- Civil Comments: Mentioned as a source for training data for DETOXIFY.
Key Insights and Ideas
- Speech data is a powerful tool for bias reduction: Accessing speech audio during inference is key to reducing false positives on samples mentioning demographic groups.
- Context is king (and speech provides it): Prosodic and contextual information in speech helps models differentiate benign group mentions from actual toxicity, especially in ambiguous or disagreement-inducing cases.
- Classifier improvement > transcription improvement: Refining the toxicity classifier itself is more impactful for reducing bias than simply improving the accuracy of the ASR transcription pipeline.
- Speech-trained models need speech at inference: If a model is trained on both speech and text, withholding speech access at inference time can worsen anti-group bias, suggesting these models might over-rely on group mentions as spurious cues.
- Ambiguity is a major challenge: Both text-only and speech-trained-but-text-only-inference models struggle with ambiguous "cannot say" or "no consensus" samples, often exhibiting higher false positive rates when groups are mentioned.
Key Results
- Improved performance: MUTOX, the model with speech access at inference, showed an increase in both precision and recall when groups were mentioned, unlike other models.
- Reduced False Positive Rate (FPR): DETOXIFY and MUTOX showed low FPRs that decreased on group mentions. In contrast, ETOX and MUTOX-ASR (trained on speech but using only text at inference) showed increased FPRs on group mentions.
- Ambiguous samples: DETOXIFY and MUTOX achieved 0% FPR on ambiguous samples, while ETOX and MUTOX-ASR showed increased FPRs.
- Specific group biases: MUTOX-ASR showed higher FPRs for samples mentioning women, White people, Black people, and Muslims compared to samples with no group mentions. MUTOX, however, showed reduced FPRs for women and 0% FPR for Muslim mentions.
- Transcription error impact: Correcting ASR transcripts improved overall model performance but had a minimal impact on the false positive rate for group mentions, indicating that classifier bias, not transcription error, is the primary driver of these specific biases.
Limitations and Future Work
- Short audio clips: The MUTOX dataset's short audio clips (2-8 seconds) likely contributed to a higher number of "cannot say" responses and made it harder to disambiguate toxicity from genuine ambiguity or quoted speech. Longer audio fragments are suggested for future work.
- Annotator bias: Annotators were discouraged from inferring speaker demographics, which might skew toxicity labels for certain dialects (e.g., AAE).
- Dataset skew: A significant portion of samples were fragments of Bible passages or religious sermons, potentially skewing the dataset.
- Limited languages: The study focused only on English and Spanish due to annotation time constraints. Future work should expand to more languages, especially under-resourced ones, where ASR performance might be a more significant factor.
- Exploratory nature: The study is considered exploratory, and statistical hypothesis testing was not included.
- Future directions: The authors suggest expanding sample size, domain diversity, and language coverage. They also highlight the need to investigate the role of speech data in toxicity detection for lower-resourced languages and the potential benefits of improving ASR pipelines in those contexts. The growing interaction with LLMs also presents new avenues for toxicity detection beyond social media.
Summary updated