TL;DR
This paper introduces MoMoE, a novel framework for AI-assisted online content moderation that uses a "mixture of experts" approach to provide scalable and transparent moderation across diverse online communities. MoMoE achieves competitive performance on unseen subreddits and generates human-readable explanations, aiming to augment rather than replace human moderators.
What Problem Does This Paper Address?
Online communities, particularly large platforms like Reddit, face a significant challenge in moderating content to enforce community norms. This burden often falls on unpaid moderators, who struggle with the sheer volume and diversity of content. Existing AI-powered moderation approaches often require extensive, community-specific fine-tuning, which is a barrier for new communities. Furthermore, these systems are often opaque, making it difficult for human moderators to understand their decisions and trust their outputs. There's a need for a moderation system that is scalable, adaptable to new communities, and provides transparency to facilitate human-AI collaboration.
Methods
MoMoE is a modular framework composed of four operators:
- Allocate: This operator determines which "expert" models are most relevant for a given comment and how much weight to give their predictions. It uses two strategies:
- Classification-based: Fine-tuned RoBERTa models predict the subreddit or norm-violation category.
- Similarity-based: Sentence embeddings are used to find similar comments in source subreddits or norm-violation categories.
- Predict: This is the core operator where specialized "expert" models make moderation decisions. These experts are fine-tuned small language models (SLMs) trained using Low-Rank Adaptation (LoRA). Two types of experts are explored:
- Community-based Experts (MoMoE<sub>community</sub>): Experts fine-tuned on data from specific source subreddits.
- Norm-violation Experts (MoMoE<sub>NormVio</sub>): Experts fine-tuned to identify specific types of norm violations.
- Aggregate: This operator combines the predictions from the selected experts. Two strategies are used:
- Dot Product: A weighted composition of expert predictions.
- Majority Vote: The most frequent prediction among the top-K experts.
- Explain: This operator uses a post-hoc LLM (GPT-4o) to generate concise, human-readable explanations for MoMoE's decisions, detailing the outcome, key norm violated, consensus among experts, and salient spans from the comment.
Datasets
The authors curated two datasets:
- D<sub>Community</sub>: Consists of subreddit, comment, context, and label (whether the comment was removed). It contains 70,000 entries from 7 source subreddits.
- D<sub>NormVio</sub>: Includes norm-violation, subreddit, comment, context, and label. Norm violation labels were generated using GPT-4o and then manually grouped into 5 broader themes. This dataset contains 81,262 entries.
The evaluation was performed on 30 unseen target subreddits.
Key Insights and Ideas
- Lightweight Expert Ensembles Outperform Generalist LLMs: MoMoE, using fine-tuned SLMs, demonstrates that specialized, modular ensembles can achieve strong performance without relying on massive, general-purpose LLMs, which are often less efficient and more prone to performance drops on specific tasks.
- Community vs. Norm-Violation Experts Trade-offs: Community-specific experts offer a wider range of performance, excelling when target communities are similar to source communities. Norm-violation experts, however, provide more consistent performance across diverse communities, suggesting that certain norm violations are universally problematic.
- Transparency is Key for Human-AI Collaboration: The "Explain" operator is crucial. By providing clear, human-readable rationales, MoMoE empowers human moderators to understand decisions, identify potential biases, and intervene effectively, shifting the paradigm from automation to augmentation.
- Modular Design for Adaptability: The modular nature of MoMoE allows for individual components to be improved or swapped out, making the framework adaptable to future advancements in NLP and flexible for different moderation needs.
Key Results
- On 30 unseen subreddits, MoMoE achieved Micro-F1 scores of 0.72 (MoMoE<sub>community</sub>) and 0.67 (MoMoE<sub>NormVio</sub>), matching or surpassing strong fine-tuned baselines.
- The best performing configuration (Mistral-based MoMoE<sub>community</sub> with classification allocation and majority vote aggregation) achieved an F1 score of 0.72.
- MoMoE models consistently outperformed off-the-shelf LLMs (GPT-4o-mini and GPT-4o), which showed significant performance drops on unseen data.
- The "Explain" operator demonstrated 100% reliability in generating explanations that accurately reflect MoMoE's decision-making trace during manual inspection.
- The framework showed robust performance even on imbalanced datasets, mimicking more realistic scenarios of low violation rates.
Limitations and Future Work
- Limited Scope of In-Domain Subreddits: The study used a fixed set of 7 source subreddits. Future work could explore how different sets or numbers of experts influence performance.
- No Multi-Agent LLM Exploration: The current framework uses single-pass SLMs. Future work could investigate multi-agent LLM systems where agents embody different roles (community, moderator persona, norm violation).
- Label Noise and Annotation Bias: The training data is derived from existing moderator actions, which can be inconsistent. LLM-generated labels for norm violations also have inherent limitations.
- English-Only Evaluation: The current evaluation is limited to English-speaking subreddits. Generalizing to multilingual or code-switched communities requires further research.
- Latency and Resource Overhead: While SLMs are lightweight, invoking multiple experts and an LLM for explanations adds latency, which could be a concern in high-traffic settings.
- Lack of User-Centric Evaluation: The study primarily relies on quantitative metrics and manual validation. Controlled user studies are needed to assess the practical utility and impact on human moderators' workflows, trust, and decision times.
Future work directions include user-centric evaluations, exploring proactive moderation (e.g., promoting prosocial content), and developing more sophisticated expert selection and dynamic adaptation mechanisms.
Summary updated