TL;DR
The paper introduces PLURULE, a novel multimodal and multilingual benchmark designed to evaluate AI models on content moderation within pluralistic online communities (like Reddit), where rules vary drastically from one group to another. The authors reframe moderation as a multiple-choice task matching comments to specific community rules, revealing that even state-of-the-art models like GPT-5.2 struggle significantly, barely outperforming a simple baseline.
What Problem Does This Paper Address?
Traditional automated content moderation treats the internet as a monolith, focusing on broadly defined global categories like toxicity, hate speech, or harassment. However, modern social media platforms increasingly rely on decentralized, community-governed structures where individual groups define their own norms. What is perfectly acceptable in one community might be a strict violation in another (for example, trading insults is encouraged in r/RoastMe but banned in most other spaces).
Existing moderation datasets fail to capture this nuance because they collapse community-specific rules into coarse, universal categories or treat moderation as a simple binary yes/no decision. This paper addresses the lack of benchmarks capable of evaluating whether AI can adapt to diverse, context-dependent local standards and simultaneously reason across a community's entire rule set.
Methods
The authors formalize pluralistic moderation as a multiple-choice question, mirroring how human moderators operate in the real world. Given a target comment and its surrounding context, the model must identify which specific community rule was violated from a list of options, or choose "No rules broken."
To build this benchmark reliably from historical data, they developed a five-phase pipeline:
- Data Collection: Extracted millions of public comments authored by human moderators from the Pushshift Reddit archives.
- Rule Matching: Used a multilingual text embedding model (
Qwen3-Embedding-8B) to match historical moderator comments to current subreddit rules using strict similarity and ambiguity thresholds. - Instance Construction: Created contrastive pairs consisting of a rule-violating thread and a compliant thread from the exact same submission to test the model's discrimination capabilities.
- Verification: Employed a large language model (
Qwen3-30B-A3B-Instruct) to verify that the moderator comment actually enforced the rule, achieving a human-validated accuracy of 96%. - Semantic Clustering: Grouped subreddits and rules into distinct categories using UMAP and HDBSCAN to enable fine-grained evaluation.
Datasets
The paper introduces the PLURULE benchmark, which contains:
- 13,371 moderation instances (each containing one violating and one compliant thread from the same submission).
- 72,675 total comments and 3,643 images, making it fully multimodal.
- Coverage across 1,989 distinct Reddit communities and 2,885 rules.
- Support for 9 languages (with at least 10 instances each).
The dataset is clustered into 25 subreddit categories (e.g., politics, gaming) and 27 rule categories (e.g., civility, spoilers, low-effort posts) to ensure substantial diversity.
Key Insights and Ideas
- Universal vs. Local Norms: Large Vision-Language Models (VLMs) are reasonably good at identifying violations of universal norms like civility (69% accuracy) and self-promotion (63% accuracy). This is likely because these concepts appear frequently across their pre-training data.
- The Context Blindspot: Models completely fall apart on rules requiring deep local context or subjective community standards. Categories like low-effort posts (43%), evidence-based claims (47%), and topical relevance (44%) all performed worse than the baseline.
- More Context Does Not Equal More Competence: Intuitively, providing a model with the entire discussion thread, the original post text, user labels, and images should help. However, adding full context only provided a marginal 2–3 percentage point boost for GPT-5.2 compared to evaluating the comment in isolation.
- Reasoning Traces Don't Fix Pluralism: Activating high-reasoning capabilities or using "thinking" model variants (like Qwen3-VL-Thinking) did not solve the problem. Models often fixate on generic incivility rather than the specific, nuanced rule being broken.
Key Results
- The Baseline: Because the dataset is balanced with 50% violating and 50% compliant comments, a trivial baseline that always guesses "No rules broken" achieves 50% accuracy.
- GPT-5.2 Performance: OpenAI's flagship model achieved a peak accuracy of just 57.7% (with high reasoning effort and full context)—barely outperforming the simple 50% baseline.
- Open-Weight Models: The Qwen3-VL family (ranging from 4B to 30B parameters) struggled even more, with most variants failing to surpass or even meet the 50% baseline under full context conditions (e.g., Qwen3-VL-30B Instruct hit 52.3%, while its Thinking variant sat at 49.5%).
- Compliance Bias: Models consistently performed better at recognizing compliant comments than identifying actual rule violations. When the evaluation metric was adjusted to prioritize violation detection (a 2:1 weighted accuracy), GPT-5.2's score dropped from 57.6% to 52.6%.
Limitations and Future Work
- Public Action Bias: PLURULE relies on public comments left by moderators. It misses content that was silently removed, private moderator communications, or shadow-bans, meaning the dataset is likely biased toward less severe violations.
- Demographic and Temporal Mismatch: The dataset is heavily dominated by English-language subreddits due to Reddit's user base. Additionally, historical comments (2005–2023) were matched against rules collected in late 2025; though semantic matching handles minor rewording, some rules may have fundamentally changed over time.
- Missing Historical User Data: Certain rule violations, such as ban evasion or tracking repeat offenders, require historical user data that cannot be collected due to privacy constraints.
- Future Directions: The authors suggest that future work should move away from zero-shot prompting and instead explore fine-tuning on community-specific examples or utilizing Retrieval-Augmented Generation (RAG) to condition models on historical local moderation decisions.
Summary updated