Whether an online comment is abusive often depends on what it replies to. "I agree" is harmless on its own and hateful under a slur. A reply that quotes a slur in order to rebut it is counterspeech, not abuse. Moderation models handle this badly. When conversation history is pasted in front of the target comment, they often get worse: the toxic words in the parent bleed into the judgment of the reply. The strongest structure-aware alternative, a graph network over reply links, handles the conversation's shape well. But it first squashes every comment into one frozen vector, so a word in the reply can never be compared directly with a word in the parent.
We propose a middle path. The reply tree around a target comment is flattened into a single token sequence that an ordinary pretrained Transformer can read. Every token also carries a learned attention bias that encodes how its comment relates to the target in the tree: the target itself, its parent, its grandparent, the thread root, or a sibling reply visible on screen. The model then has full word-to-word attention across the thread, plus an explicit structural signal it can use to decide how far up the tree to look. The specific bet is that this relation-to-target signal is what fixes "contamination" (false alarms on counterspeech and benign replies under toxic parents). The bet is that plain text markers and chronological order do not fix it. The experiments are built so that bet can fail cleanly.
Core Claim
Two directions meet here. Tree linearization is the craft of turning a tree into a flat token sequence without losing which node hangs under which. In context-dependent toxicity detection, a comment's label depends on the conversation that precedes it. The conversation side supplies the problem and the evidence of a gap. The linearization side supplies the mechanism.
The gap is well documented. Introducing CAD: the Contextual Abuse Dataset builds a Reddit abuse dataset in which 25–32% of abusive posts need prior posts to be identified. On that dataset BERT's recall on identity-directed abuse falls from 60.1% to 25.3% when context is required, and counterspeech F1 stays below 0.1. Humans Need Context, What about Machines? compares ways of feeding context to a classifier on CAD. It finds that naive concatenation lowers Macro F1 from 71.10 (no context) to 69.40, while compressing the context into a single "super-token" vector (Context-Embed) raises it to 73.04. Graphically Speaking models Reddit threads as reply graphs, runs a graph attention network over BERT comment embeddings, and trims each thread to what a user actually sees on screen. It reaches 76.24% F1 on a balanced CAD subset and 74.07% accuracy on context-sensitive samples. It also reports a "three-hop sweet spot": ancestry beyond three levels adds noise, while visible sibling replies add framing.
Together these results say that the tree's shape carries the useful signal, and that losing it, as concatenation does, actively hurts. No existing system combines that shape with word-level attention between a reply and its ancestors.
Our claim is narrower than "linearize the tree." Prior linearizations mostly order nodes from the root, using depth-first traversal with brackets or backtracking tokens, as in Grammar as a Foreign Language and Constrained Sequence-to-Tree Generation for Hierarchical Text Classification. Classifying one comment is a different job. What matters is each other comment's relation to the target, and the model should be able to weigh relations differently in different attention heads. One head might attend to the parent for direct insults, another to the root for stereotypes that the root sets up. That is the "dynamic scope" problem the conversation literature leaves open.
Research Program
We build three things.
The first is a target-centered linearization. The thread is trimmed with the on-screen visibility rule from Graphically Speaking: root, top-ranked replies, and the ancestral path to the target. The target comment comes first, followed by its ancestors in order of increasing distance, then the visible siblings of each ancestor. Each comment is preceded by a short marker token naming its relation, such as [PARENT], [ANC-2], [ROOT], or [SIB-OF-ANC-1]. Speaker identity is marked only by whether the author matches the target's author, since self-replies and pile-ons look different.
The second is a relation-to-target attention bias. Let $\rho(i)$ be the tree relation, relative to the target, of the comment containing token $i$. This is a pair (hops up from the target, sibling offset at that level), in the spirit of the two-dimensional child-index coordinates in Rethinking Positional Encoding in Tree Transformer for Code Representation. Each attention head $h$ computes
$$a^{h}_{ij} = \frac{q^{h}_i \cdot k^{h}_j}{\sqrt{d}} + \beta_h\big(\rho(i), \rho(j)\big),$$
where $q, k$ are the usual query and key vectors, $d$ is the head dimension, and $\beta_h$ is a small learned table indexed by the pair of relations. Ordinary token positions are kept within each comment and reset between comments, so chronological distance in the flattened string no longer stands in for conversational distance. The code-modeling paper showed that one-dimensional positions fail to capture tree shape in syntax trees. We reuse its coordinate idea, but anchor coordinates at the classification target rather than the root, and we test whether the idea carries over to conversation threads.
The third is a contamination test set built from CAD. We take targets labeled counterspeech or neutral whose parent is abusive. For each, we construct swaps in which the same target sits under a benign parent from elsewhere in the corpus. For context-dependent abusive targets such as "I agree," we construct the reverse swap. A model that has learned the conversational relation should change its prediction where the label truly depends on context, and only there. Swapped examples cannot have gold labels for every case. So we use the swaps only where the CAD annotation states that the "Current" post alone was sufficient, or that "Previous" context was needed, and we hand-check a sample of 300.
Analysis then asks which heads learn which relations. For each head we inspect the learned $\beta_h$ tables. We ask whether stereotype errors on the French data shrink when root-attending heads exist. And we check whether gains rise with thread depth, which would be evidence that structure, rather than extra parameters, is doing the work. We are mindful of Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion, which shows that Transformers learn depth-specific shortcuts rather than true recursion. This is why we hand the model explicit relation features instead of hoping it infers relations from brackets. It is also why we test on threads deeper than any seen in training.
Baselines
The naive reference points are a target-only classifier, using the same pretrained encoder with no context, and chronological concatenation, which joins context and target with separator tokens. On CAD, Humans Need Context, What about Machines? reports 71.10 and 69.40 Macro F1 for these two. We rerun both on our splits with the same backbone. Any proposed model that fails to beat target-only is a negative result for the whole approach.
The strongest compressed-context baseline is Context-Embed from the same paper: a first Transformer pools the context into one vector that a second Transformer reads alongside the target. It reports 73.04 Macro F1 on CAD. It is the best published sequence-style method and the most direct test of whether word-level attention across comments matters. For the French stereotype data, the strongest reported system is that paper's BART encoder-decoder, with context in the encoder and the target in the decoder, at 49.71 F1 on $FR_{stereo}$ versus 28.40 for concatenation. We rerun it as well.
The strongest structure-aware baseline is the 3-layer graph attention network with visibility trimming from Graphically Speaking, at 76.24% F1 on its balanced CAD subset (3,605 abusive, 3,605 non-abusive) and 74.07% accuracy on context-sensitive samples. To compare fairly, we use their balanced subset construction and trimming rule for every system, the same encoder family for node embeddings and for our backbone, and an identical hyperparameter search budget (the same number of trials, the same seeds) for every system. Beating it means exceeding its F1 on the balanced subset and its context-sensitive accuracy, with paired bootstrap confidence intervals over five seeds that exclude zero. We will report both our rerun and the paper's published figure.
Two ablations isolate the key ingredient. The first, markers-only, uses the same target-centered ordering and relation marker tokens but sets $\beta_h = 0$. The second, bias-only, keeps the bias but removes the marker tokens. A third variant, root-ordered bracketed traversal, linearizes the same trimmed tree by root-first depth-first traversal with brackets, as in classic linearization, to test whether anchoring at the target matters.
Why not just…?
Why not just concatenate the ancestral path with speaker and depth tags? This is the real threat. A long-context encoder reading "[ROOT] … [PARENT] … [TARGET] …" might recover most of the gain, and if so, the attention bias is unnecessary. This is exactly the markers-only ablation. If it matches the full model on overall F1 and on the contamination set, we will report that marker tokens are sufficient. That would still be a useful, cheap result relative to graph pipelines. Our prediction is that it closes much of the gap on overall F1 but not on counterspeech under abusive parents, because tags do not stop the parent's abusive words from receiving attention. Only a learned, relation-specific bias can turn that attention down per head.
Why not just use the graph network? It already reaches 76.24% F1. But it pools each comment into a frozen vector before any comment-to-comment interaction happens. Counterspeech that quotes a slur in order to rebut it looks lexically like the abuse it answers. Telling them apart requires comparing the quoted span with its framing words against the parent, and pooling removes exactly that information. The contamination set is where we expect the difference to show. If the graph model does as well there, the word-level argument fails.
Why not just prompt a frontier LLM with the raw thread? Can LLMs Be Efficient Predictors of Conversational Derailment? reports strong prompted results on derailment forecasting, though no number for CAD-style abuse classification was found on Lacuna. Latency and cost matter for moderation. Implanting LLM's Knowledge via Reading Comprehension Tree reports 0.14s per item with LLM calls versus 0.01s for small models. We will still run a zero-shot and a few-shot prompted LLM on the balanced CAD test set and the contamination set, as an upper reference rather than a competitor at equal compute. If it dominates on contamination, our method's case reduces to efficiency, and we will say so.
Why not just add the tree as an auxiliary loss, as in dialogue disentanglement? Conversation- and Tree-Structure Losses for Dialogue Disentanglement reads a flat chronological stream and uses tree distance only in its training loss. It reports trouble when parents are far away in the stream, which is the failure mode our relation-reset positions are designed to remove. We include a variant with chronological input plus a tree-distance auxiliary loss to test this directly.
Proposed Experiments
The primary benchmark is CAD, evaluated both on the balanced subset from Graphically Speaking and on the original six-way split from Introducing CAD, where we report counterspeech F1 separately. The secondary benchmark is $FR_{hate}$/$FR_{stereo}$ from Humans Need Context, What about Machines?, where 68% of stereotypes need context. That dataset tests whether root-attending heads help with stereotypes framed at the top of a thread. The Turkish tweet-reply data from A Dataset for Investigating the Impact of Context, whose trees are only one level deep, serves as a sanity check: our method should match concatenation there (BERTurk reports 73.57 F1 with context), since there is no structure to exploit.
The study hinges on three decisive checks. First, on the contamination set, the full model should cut the false-positive rate on counterspeech and benign replies under abusive parents relative to concatenation, Context-Embed, the graph network, and markers-only, without lowering recall on context-dependent abuse. Second, when CAD is binned by trimmed-thread depth, the full model's advantage over markers-only should grow with depth. On held-out threads deeper than any seen in training, performance should degrade more gently than root-ordered bracketing does, which would be evidence against shallow depth shortcuts. Third, after training, the bias tables $\beta_h$ should show heads that specialize to parent versus root relations. Zeroing root-specialized heads should hurt $FR_{stereo}$ more than CAD person-directed abuse.
If none of these checks favors the bias over marker tokens, the honest conclusion is that a plain text marking of the tree is sufficient for this task. That would be a cheaper result than any proposed here, and worth publishing.
References
- Introducing CAD: the Contextual Abuse Dataset
- Humans Need Context, What about Machines? Investigating Conversational Context in Abusive Language Detection
- Graphically Speaking: Unmasking Abuse in Social Media with Conversation Insights
- Grammar as a Foreign Language
- Constrained Sequence-to-Tree Generation for Hierarchical Text Classification
- Rethinking Positional Encoding in Tree Transformer for Code Representation
- Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion
- Can LLMs Be Efficient Predictors of Conversational Derailment?
- Implanting LLM's Knowledge via Reading Comprehension Tree for Toxicity Detection
- Conversation- and Tree-Structure Losses for Dialogue Disentanglement
- A Dataset for Investigating the Impact of Context for Offensive Language Detection in Tweets