Detect Profane Language in Streaming Services to Protect Young Audiences

Jingxiang Chen, Kai Wei, Xiang HaoACL 2021

TL;DR

This paper tackles the challenge of detecting profane language in streaming services, which is crucial for protecting young audiences. They introduce a novel data pipeline that effectively handles long video captions and integrates with a multi-head self-attention model, achieving significant accuracy improvements with fewer parameters compared to existing methods.

What Problem Does This Paper Address?

The proliferation of streaming services like Netflix and Prime Video has led to increased exposure to online content, raising concerns about the presence of profane language, especially for younger viewers. Detecting profanity in streaming services is particularly challenging because video captions are often very long, presenting a hurdle for traditional natural language processing models that struggle with long sequences. While previous research has focused on deep learning techniques for long sentences, this paper highlights a gap in improving the data pipelines themselves, which are critical for effective profanity detection in this domain. Furthermore, the data available for streaming services differs from typical social media text, requiring specialized approaches.

Methods

The core of this work lies in their proposed data pipeline and a custom multi-head self-attention model.

Data Pipeline: * Data Collection: They collected 150,000 titles from Prime Video and augmented this with 5,010 titles from the MPAA database. * Annotation: A detailed codebook based on MPAA and TVPG guidelines was developed, and 15 human annotators labeled video titles with profanity levels (None, Mild, Moderate, Strong, Severe). * Sentence Label Generation: To address the long caption issue, they developed a method to generate sentence-level labels. This involved identifying unambiguous keywords and using a pre-defined threshold to categorize sentences containing them as profane. To mitigate noise from this rule-based approach, they employed a knowledge distillation-like technique where an intermediate classifier generated "soft targets" for training. * Data Augmentation: Beyond the MPAA data, they used active learning to manually label sentences that the model struggled with, further refining the training data.

Multi-Head Self-Attention Model: * The model first embeds words using GloVe. * A bidirectional GRU (BiGRU) layer captures long-term word dependencies. * This is followed by a multi-head self-attention layer, allowing the model to weigh different keywords within a sentence flexibly. * A fully connected layer with a sigmoid activation function produces the final prediction. * Crucially, the model is trained using the "soft targets" generated by the sentence label generation process, making it more robust to label noise.

Datasets

  • Prime Video (PV) Data: 150,000 titles collected from the streaming service, forming the primary dataset.
  • MPAA Database: 5,010 additional titles collected and adapted from the Motion Picture Association of America database.
  • Human Annotation: 15 annotators were employed to label the collected data.

The paper introduces a newly curated dataset for profanity detection in streaming service titles and their corresponding captions, which differs from existing datasets primarily focused on social media text.

Key Insights and Ideas

  • Data Pipeline is Key: The paper strongly emphasizes that improving data collection and processing pipelines is as crucial as developing sophisticated models for tasks like profanity detection, especially when dealing with long sequences.
  • Sentence-Level Labels for Long Captions: Generating sentence-level labels from title-level information is an effective strategy to handle the challenge of extremely long video captions.
  • Soft Targets for Noise Reduction: Using soft targets from an intermediate model, inspired by knowledge distillation, significantly improves robustness against the inherent noise in automatically generated labels.
  • Active Learning for Refinement: Active learning proves valuable for identifying and correcting model weaknesses, particularly at the boundaries of profanity levels.
  • Context Matters for Ambiguous Keywords: The self-attention mechanism's ability to weigh words based on context is vital for correctly interpreting ambiguous keywords, which rule-based systems often miss.

Key Results

  • The proposed multi-head self-attention model, integrated with their data pipeline, achieved 12.5% relative accuracy improvement over the state-of-the-art distilBERT model on profane language detection.
  • Remarkably, their model required only 3% of the parameters compared to distilBERT.
  • Models built with their Generating Sentence Labels (GSL) pipeline consistently outperformed those without it, showing improvements in accuracy and recall across various precision thresholds.
  • Data augmentation was shown to be beneficial, with models using it achieving higher accuracy than those trained solely on the targeted streaming service data.
  • The self-attention model achieved the best performance in detecting the level of profanity, outperforming rule-based methods and DistilBERT.

Limitations and Future Work

  • Limitations:
    • The generated sentence labels can still contain noise due to the imperfect nature of the generation process.
  • The model's performance on ambiguous keywords can be suboptimal, as seen with DistilBERT misclassifying negative sentences containing "suck" as positive.
  • Error analysis revealed that title models might assign high weights to words related to violence or other content descriptors, potentially diluting the focus on profanity keywords.
  • The confusion matrix shows errors in overrating None and Mild titles, with some confusion between Mild and Moderate levels, even for human annotators.
  • Future Work:
    • Exploring how the proposed pipeline can be extended to detect more general profanity across different modalities, such as visual and audio content.

Summary updated

BibTeX
@inproceedings{chen2021detect,
  author = {Jingxiang Chen and Kai Wei and Xiang Hao},
  title = {{Detect Profane Language in Streaming Services to Protect Young Audiences}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure