Community Specific Norm Enforcement in Automated Moderation

This direction is split into other directions: context-aware moderation. Its last version stays readable and citable here.

Version 1 of 135 papers54 conceptsfrozen 2026-10-04

For years, the standard approach to automated content moderation has been "universal toxicity detection." Tools like Google’s Perspective API or OpenAI’s moderation endpoint are trained to identify a global baseline of "badness"—hate speech, profanity, and overt harassment. However, as researchers like David Jurgens and Eshwar Chandrasekharan argue in A Just and Comprehensive Strategy for Using NLP to Address Online Abuse, this norm-agnostic approach is fundamentally flawed.

The reality of the internet is that "acceptability" is not a universal constant; it is a moving target defined by the specific micro-culture of a community. A term of endearment in one subreddit might be a bannable offense in another. Recent research has shifted away from building better "hate speech detectors" toward building "norm-aware" systems that can reason about the specific rules of the space they are policing.

The Toxicity Gap

The most striking finding in this research area is the "functionality gap" between what AI models detect and what human moderators actually do. In Toxicity Detection is NOT all you Need, Yang Cao and Hal Daumé III reveal that standard toxicity filters cover less than 20% of the rules enforced by volunteer moderators.

Most moderation effort is spent on community-specific guidelines—such as topical relevance, source reliability, or "low-effort" post restrictions—that have nothing to do with toxicity. For example, a finance forum might ban "speculative claims," while a gaming community might ban "spoilers." A universal model trained on broad social media data is blind to these distinctions, leading to a "one-size-fits-all" failure where niche communities are either under-protected or over-censored.

Grounding Moderation in Explicit Rules

To bridge this gap, researchers are moving toward Policy-Aligned Moderation (PAM). Instead of asking a model "Is this text toxic?", these systems ask "Does this text violate this specific rule?"

The CPL-NoViD framework, developed by Jonathan May and Zihao He, uses prompt-based learning to feed the actual text of a community's rules directly into the model. By framing moderation as a natural language question grounded in the rulebook, the model can generalize to new communities without needing thousands of new labeled examples. Similarly, the Community Rules-based Content Moderation (CRCM) framework Let Community Rules Be Reflected in Online Content Moderation uses topic modeling to map posts to specific community guidelines, allowing for interpretable decisions rather than "black-box" scores.

Capturing the Unwritten Rulebook

Not all rules are written in a sidebar. Many communities operate on "unwritten rules" or "macro norms"—universal but unspoken standards of behavior. Eric Gilbert and Eshwar Chandrasekharan identified eight such "macro norms" by analyzing millions of deleted Reddit comments in Hybrid Approaches to Detect Comments Violating Macro Norms on Reddit.

To capture these subtle signals, researchers like Yong-Woo Kim have introduced the Partial Attention Transformer (PAT). This model Decoding the Rule Book: Extracting Hidden Moderation Criteria from Reddit Communities extracts hidden moderation criteria as interpretable score tables, revealing how different subreddits uniquely define "incivility." For instance, some groups may have a high tolerance for aggressive debate but a zero-tolerance policy for "second-person framing" (e.g., "You are wrong") in personal attacks.

Contextual and Temporal Awareness

A single comment is rarely enough to determine a violation. Context matters. In Analyzing Norm Violations in Live-Stream Chat, Jihyung Moon and Dong-Ho Lee found that incorporating just two minutes of chat history in a live stream like Twitch can boost moderation performance by 35%.

This contextual anchoring is also spatial. The One-Model-Connects-All (OMCA) framework One-Model-Connects-All: A Unified Graph Pre-Training Model for Online Community Modeling represents users, communities, and text as a single graph. This allows the model to "know" that a specific user is posting in a specific sub-community, providing the social environment necessary to interpret whether a reclaimed slur is a sign of camaraderie or a violation of local norms.

Efficiency through Specialization

While Large Language Models (LLMs) like GPT-4 are powerful, they are often too slow and expensive for real-time moderation. More importantly, they are "averaged" over the entire internet, which dilutes their ability to understand niche norms.

In SLM-Mod: Small Language Models Surpass LLMs at Content Moderation, Xianyang Zhan and Koustuv Saha demonstrate that Small Language Models (SLMs) under 15B parameters actually outperform GPT-4 when fine-tuned on community-specific data. By specializing on local norms, these smaller models achieve higher recall and accuracy, proving that in the world of community governance, local expertise beats global scale.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_community_specific_norm_enforcement_in_a_0642f04f,
  title = {Community Specific Norm Enforcement in Automated Moderation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/community-specific-norm-enforcement-in-automated-moderation/rdr_0642f04f453348f287d58c9d8c53678f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_community_specific_norm_enforcement_in_a_0642f04f_v1,
  title = {Community Specific Norm Enforcement in Automated Moderation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/community-specific-norm-enforcement-in-automated-moderation/rdr_0642f04f453348f287d58c9d8c53678f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}