Introduction
Automated content moderation is a critical component of modern social platforms, yet traditional binary classifiers often struggle with the dynamic nature of policy enforcement. These "black-box" models typically require extensive re-labeling and retraining whenever a policy is updated. Furthermore, they often fail to capture the nuance of complex policies or the "long tail" of rare violations.
To address these challenges, researchers at LinkedIn have introduced QnA-CM (Question Answering based Content Moderation). This system reformulates content moderation as a binary question-answering task, breaking down complex policies into modular, interpretable components.
The QnA-CM Framework
Instead of training a model to output a single "Spam" or "Not Spam" label, QnA-CM decomposes a policy into specific "themes." For each theme, a natural language question is generated. A fine-tuned Large Language Model (LLM) then analyzes the content to answer these questions with a simple "Yes" or "No."
The system consists of three primary layers:
- Policy Decomposition: Policies are translated into a set of binary questions (e.g., "Does this content mention cryptocurrency?").
- Binary Q&A: An LLM (such as BERT or DeBERTa) is fine-tuned to act as the answering engine.
- Decision Logic: A human-defined logic layer uses Boolean operators (AND, OR, NOT) to aggregate the LLM's answers into a final moderation verdict.
Scalable Multi-level Inference
To maintain computational efficiency in production, QnA-CM employs a two-level inference strategy.
As shown in the figure above, Level 1 (L1) first evaluates a "common intent" question (e.g., "Is this content attempting to sell something?"). If the answer is negative, the content is cleared immediately. Only if L1 returns a positive result does the system proceed to Level 2 (L2), where it evaluates specific policy themes. This hierarchical approach ensures that the computational cost remains comparable to standard binary classifiers.
Key Advantages
- Zero-Shot Adaptability: By using natural language prompts, the model can leverage its pre-trained semantic knowledge. This allows it to identify new violation types immediately after a policy update, even without specific training data for the new theme.
- Decoupled Logic: Policy changes can be implemented "on the fly" by updating the decision logic or adding new questions, eliminating the need for full model retraining.
- Semantic Disentanglement: The targeted nature of the questions helps the model distinguish between general sentiment and specific policy violations. For instance, in hate speech detection, it can better differentiate between generally rude content and content that specifically targets protected inherent traits.
Experimental Results
The system was evaluated on two large-scale LinkedIn datasets: one for Commercial Spam (CS) and another for Hate Speech. The primary metric used was Recall at 95% Precision, which is vital for minimizing false positives in production environments.
- Performance Gains: On the Commercial Spam task, QnA-CM improved recall to 48.17%, up from the baseline's 41.36%.
- Agility: When simulating policy changes (introducing new themes), the baseline's recall dropped significantly to 11.38%. In contrast, QnA-CM's zero-shot capabilities allowed it to maintain a much higher recall of 33.68%.
- Precision in Hate Speech: The system reduced the False Positive Rate (FPR) for "hateful but non-violating" content from 0.71 to 0.40, demonstrating superior nuance in policy enforcement.
Conclusion
QnA-CM represents a significant shift from monolithic classification to modular, interpretable content moderation. By treating policy enforcement as a Q&A task, platforms can achieve higher accuracy and respond more rapidly to the evolving landscape of online safety. Future work will explore the use of even larger generative models and the automation of question generation from legal policy documents.
Summary updated