Content Moderation for Evolving Policies using Binary Question Answering

Sankha Subhra Mullick, Mohan Bhambhani, Suhit Sinha, Akshat Mathur, Somya Gupta, Jidnya Shah

Introduction

Automated content moderation is a critical component of modern social platforms, yet traditional binary classifiers often struggle with the dynamic nature of policy enforcement. These "black-box" models typically require extensive re-labeling and retraining whenever a policy is updated. Furthermore, they often fail to capture the nuance of complex policies or the "long tail" of rare violations.

To address these challenges, researchers at LinkedIn have introduced QnA-CM (Question Answering based Content Moderation). This system reformulates content moderation as a binary question-answering task, breaking down complex policies into modular, interpretable components.

The QnA-CM Framework

Instead of training a model to output a single "Spam" or "Not Spam" label, QnA-CM decomposes a policy into specific "themes." For each theme, a natural language question is generated. A fine-tuned Large Language Model (LLM) then analyzes the content to answer these questions with a simple "Yes" or "No."

Overview of the QnA-CM system architecture and policy update workflow.

The system consists of three primary layers:

  1. Policy Decomposition: Policies are translated into a set of binary questions (e.g., "Does this content mention cryptocurrency?").
  2. Binary Q&A: An LLM (such as BERT or DeBERTa) is fine-tuned to act as the answering engine.
  3. Decision Logic: A human-defined logic layer uses Boolean operators (AND, OR, NOT) to aggregate the LLM's answers into a final moderation verdict.

Scalable Multi-level Inference

To maintain computational efficiency in production, QnA-CM employs a two-level inference strategy.

Multi-level inference architecture (L1 and L2) for efficient content moderation.

As shown in the figure above, Level 1 (L1) first evaluates a "common intent" question (e.g., "Is this content attempting to sell something?"). If the answer is negative, the content is cleared immediately. Only if L1 returns a positive result does the system proceed to Level 2 (L2), where it evaluates specific policy themes. This hierarchical approach ensures that the computational cost remains comparable to standard binary classifiers.

Key Advantages

  • Zero-Shot Adaptability: By using natural language prompts, the model can leverage its pre-trained semantic knowledge. This allows it to identify new violation types immediately after a policy update, even without specific training data for the new theme.
  • Decoupled Logic: Policy changes can be implemented "on the fly" by updating the decision logic or adding new questions, eliminating the need for full model retraining.
  • Semantic Disentanglement: The targeted nature of the questions helps the model distinguish between general sentiment and specific policy violations. For instance, in hate speech detection, it can better differentiate between generally rude content and content that specifically targets protected inherent traits.

Experimental Results

The system was evaluated on two large-scale LinkedIn datasets: one for Commercial Spam (CS) and another for Hate Speech. The primary metric used was Recall at 95% Precision, which is vital for minimizing false positives in production environments.

  • Performance Gains: On the Commercial Spam task, QnA-CM improved recall to 48.17%, up from the baseline's 41.36%.
  • Agility: When simulating policy changes (introducing new themes), the baseline's recall dropped significantly to 11.38%. In contrast, QnA-CM's zero-shot capabilities allowed it to maintain a much higher recall of 33.68%.
  • Precision in Hate Speech: The system reduced the False Positive Rate (FPR) for "hateful but non-violating" content from 0.71 to 0.40, demonstrating superior nuance in policy enforcement.

Conclusion

QnA-CM represents a significant shift from monolithic classification to modular, interpretable content moderation. By treating policy enforcement as a Q&A task, platforms can achieve higher accuracy and respond more rapidly to the evolving landscape of online safety. Future work will explore the use of even larger generative models and the automation of question generation from legal policy documents.

Summary updated

Abstract

Content moderation on social media is governed by policies that are intricate and frequently updated with evolving world events. However, automated content moderation systems often restrict easy adaptation to policy changes and are expected to learn policy intricacies from limited amounts of labeled data, which make effective policy compliance challenging. We propose to model content moderation as a binary question answering problem where the questions validate the loosely coupled themes constituting a policy. A decision logic is applied on top to aggregate the theme-specific validations. This way the questions pass theme information to a transformer network as explicit policy prompts, that in turn enables explainability. This setting further allows for faster adaptation to policy updates by leveraging zero-shot capabilities of pre-trained transformers. We showcase improved recall for our proposed method at 95\% precision on two proprietary datasets of social media posts and comments respectively annotated under curated Hate Speech and Commercial Spam policies.
BibTeX
@inproceedings{mullick2023content,
  author = {Sankha Subhra Mullick and Mohan Bhambhani and Suhit Sinha and Akshat Mathur and Somya Gupta and Jidnya Shah},
  title = {{Content Moderation for Evolving Policies using Binary Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-industry.54},
}
Figure