Imagine a content moderation system that flags a social media post with a score of "0.95 Toxic." What does that number actually mean? Is the post an explicit violent threat, a casual swear word, a subtle gender stereotype, or a coordinated piece of political defamation?
For years, natural language processing treated online toxicity as a monolithic, binary attribute: text was either clean or toxic. However, a growing body of research recognizes that online harm is multi-faceted, hierarchical, and deeply context-dependent. To build effective moderation tools, design precise interventions, and provide transparent explanations, models must move beyond binary detection toward fine-grained categorization.
The Monolithic Fallacy
Treating all offensive content as a single axis of "badness" creates a fundamental blind spot. Lees et al. demonstrated that general toxicity metrics are frequently insufficient for identifying specific categories of online harm, such as misogyny or aggressiveness. In fact, a generic toxicity score can sometimes be counter-indicative of specialized harms, failing to flag subtle but deeply damaging behavior.
To resolve this, researchers argue that we must change how models represent toxicity internally. Cho et al. proposed representing harmful content as a set of distinct semantic categories rather than a single monolithic attribute. By breaking toxicity down into discrete vectors, models can be steered more precisely during detoxification, allowing them to catch rare or nuanced types of harm that generalized systems routinely overlook.
The Hierarchy and Spectrum of Harm
Fine-grained classification typically operates by organizing toxic behavior into multi-tiered hierarchies. In the domain of online sexism, for example, a system might first determine if a post is sexist (binary), then classify its broad category (e.g., derogation versus prejudiced discussion), and finally map it to a highly specific behavioral vector (e.g., tone policing or objectification).
As Samani et al. note, socially harmful content exists on a continuous spectrum ranging from subtle derogation to explicit threats. Capturing this spectrum requires models to understand the hierarchical relationships between broad categories of harm and their specific, fine-grained manifestations.
However, increasing the granularity of a taxonomy introduces a steep performance penalty. Mahmoudi highlights that while binary detection is relatively straightforward, building effective models for granular subtasks is exceptionally difficult due to severe data scarcity and increased class granularity. This performance drop is a recurring theme in recent benchmarks: Das et al. and Andrew both observe that while standard machine learning models achieve high accuracy on binary classification, their performance degrades significantly when forced to distinguish between highly specific sub-categories of hate speech.
Why Fine-Grained Classification is Hard
The core difficulty of fine-grained classification stems from the fact that categories of harm are rarely mutually exclusive. Instead, they feature overlapping linguistic, stylistic, and psychological dimensions.
Overlapping Categories
A post can be aggressive without being misogynistic, or it can be deeply misogynistic through subtle belittling without using any aggressive language. Muti et al. explored this intersection, demonstrating that models must capture fine-grained linguistic nuances to successfully untangle misogyny from general aggressiveness. Because these dimensions frequently co-occur, Gajo et al. leverage multi-label classification objectives to simultaneously identify distinct categories of hate speech within a single text, capturing overlapping dimensions of prejudice like racism and misogyny.
Shared Linguistic Features
Even outside of identity-based hate, different types of problematic content share structural similarities. Dinkov et al. point out that nuanced categories like sensationalism, conspiracy theories, and defamation are difficult to separate because they often exhibit overlapping linguistic and stylistic features. A model cannot rely on simple keyword matching; it must understand the underlying rhetorical strategy.
Disentangling Target and Intent
To achieve true granularity, modern frameworks separate what is being said from who is being targeted and how the attack is delivered.
Muhammad et al. advocate for target-aware annotation, a strategy where content is labeled not just by its general toxicity, but by the specific protected attributes or groups being targeted (e.g., ethnicity, religion, or gender). This representational strategy allows models to distinguish between general profanity and targeted hate speech.
By explicitly separating the target from the mechanism of the attack, models gain a much more actionable understanding of the text. As Zia et al. show, classifying content by both the targeted group and the specific attack type (such as slurs versus contempt) yields a nuanced diagnostic mapping. This shifts the paradigm of content moderation from merely filtering "bad" text to understanding the specific social dynamics of online harm.
Key Foundational Work
Detecting Misogynous Tweets (2018) — contributed 1 concept elements to this direction.