'Beach' to 'Bitch': Inadvertent Unsafe Transcription of Kids' Content on YouTube

Krithika Ramesh, Ashiqur R. KhudaBukhsh, Sumeet KumarAAAI 2022 · DOI

Introduction

Content moderation for children typically focuses on filtering out inappropriate source material. However, a new study highlights a "downstream" risk: the inadvertent generation of profanity and taboo words by Automatic Speech Recognition (ASR) systems used for accessibility and literacy. This phenomenon, termed "inappropriate content hallucination," occurs when AI tools misinterpret innocent audio—such as "beach" or "crab"—as offensive language like "bitch" or "crap."

Example of an ASR hallucination where "crab" is transcribed as "crap" in a children's video.

As YouTube Kids grows, millions of children rely on these auto-captions. Introducing inappropriate vocabulary to an impressionable audience through technical errors represents a significant safety gap in current AI pipelines.

Methodology and Datasets

To investigate the scale of this issue, researchers developed a multi-step pipeline to detect and correct hallucinations. They compared transcripts from two major commercial ASR systems: Google-based YouTube internal captions and Amazon Transcribe.

The study introduced several novel resources:

  • YouTube Kids (YTK) Dataset: 7,013 videos from 24 popular channels (e.g., Cocomelon).
  • Kids' Taboo Lexicon: A curated list of 1,301 words deemed inappropriate for children, refined by removing common words found in G-rated Disney and Pixar movie subtitles.
  • $D_{taboo}$ Benchmark: A dataset of 652 verified audio clips where ASR systems hallucinated taboo words.

To verify these errors, human annotators listened to the source audio to ensure the taboo words were indeed absent. Finally, the team tested a "corrective cloze task" using Masked Language Models (MLMs) like BERT and Megatron-LM to predict the correct word based on surrounding context.

The Challenge of High-Confidence Errors

A critical finding of the research is that ASR systems often produce these hallucinations with extremely high confidence scores. The systems are not "unsure" of what they heard; rather, their internal models strongly, yet incorrectly, favor a taboo interpretation.

Distribution of ASR confidence scores for hallucinated words, showing a significant peak at high confidence (near 1.0).

These errors are frequently triggered by "baby talk," background music, or non-native English speakers. The resulting acoustic ambiguity causes the ASR to default to common patterns found in its training data—which is predominantly adult-centric (news, Wikipedia, etc.). This "adult bias" extends to the Language Models used for correction; for instance, BERT may prefer "porn" over "corn" because the former appears more frequently in its general-purpose training corpus.

Key Results and Correction Performance

The prevalence of this issue is startling: nearly one in ten videos analyzed contained at least one highly inappropriate hallucinated word. Jarring examples documented include "brave" becoming "rape," "venus" becoming "penis," and "buster" becoming "bastard."

While correcting these errors automatically is difficult, the study found that using Megatron-LM with Levenshtein distance for candidate generation fixed approximately 25-28% of errors. While the fully automated fix rate is modest, the "Top-10" accuracy (P@10) reached over 45%, indicating that these models could serve as powerful tools for human moderators to clean up transcripts.

Limitations and Future Outlook

Several hurdles remain for securing children's content:

  1. Contextual Incoherence: Many kids' videos feature repetitive or nonsensical dialogue, making it hard for LMs to use context for correction.
  2. Data Mismatch: There is a dire need for ASR and LM tools trained specifically on children's speech and safe-content corpora rather than adult-centric data.
  3. Multimodal Integration: Future systems should incorporate video signals. If a "crab" is visible on screen, the AI should be able to override an ASR system that thinks it heard "crap."

Ultimately, the study underscores a disconnect between platform safety tiers. Hallucinated captions often appear on the main YouTube site even for content intended for kids, highlighting the need for tighter safety integration across the entire ecosystem.

Summary updated

BibTeX
@inproceedings{ramesh2022beach,
  author = {Krithika Ramesh and Ashiqur R. KhudaBukhsh and Sumeet Kumar},
  title = {{'Beach' to 'Bitch': Inadvertent Unsafe Transcription of Kids' Content on YouTube}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21470},
}
Figure