Typed Syntax as the Site of Inversion: Relation-Aware Attention for Ironic Hate

Much of the online hate that slips past moderation never uses a slur. It praises, agrees, or jokes, and the harm lies in how the praise is attached to a group. The positive word modifies the wrong thing, or a group is made the grammatical subject of a mock-compliment. Sarcasm detectors and implicit-hate detectors both fail on this material, for the same reason: they decide from which words are present, not from how those words are connected. This proposal asks whether the type of grammatical link can supply the missing signal. A type is the label on the arrow between two words in a dependency parse (a parse is a tree linking each word to the word it grammatically depends on, with labels such as "subject of," "modifier of," "negation of"). The question is whether the type of link between a group mention and an evaluative word tells ironic contempt apart from an innocent sentence that merely mentions a group.

The concrete plan is a small attention module added on top of a standard pretrained encoder. Attention is restricted to words that are near each other in the parse tree. It receives a learned bias that depends on the labeled path between two words, and a separate bias for paths that start at a candidate target of the evaluation. We train it on sarcasm and implicit-hate data. The test is sharp: does the typed-path signal help specifically on ironic hate and on benign sentences that mention identity terms, beyond what the tree connectivity alone provides? Recent evidence says connectivity usually does all the work, so the typed component must earn its place against that.

Core Claim

Our claim is that in sarcastic and ironic toxicity, the inversion is localized in a small number of typed relations: an evaluative modifier or predicate linked to a target through subject, object, or modifier arcs. It is not spread over the sentence. If that is true, a model that conditions attention on typed parse paths anchored at the target should show three effects relative to a model that uses the same parse without labels. First, it should improve on the Irony subclass of the Implicit Hate Corpus. Second, it should transfer better from sarcasm-only training data to ironic hate. Third, it should make fewer false positives on neutral text containing identity terms, without any explicit anti-bias regularizer. If typed paths give no gain over untyped masking on these slices, the claim is false, and we will report that as the result.

This claim sits in a specific tension within the file. The relation-aware sparse attention study What Structural Inductive Bias Helps Transformers Reason Over Knowledge Graphs? ablated four structural ingredients over knowledge graphs. Restricting attention to graph neighbors recovered most of the gain (for example +72.5 percentage points on 3-hop MetaQA). Relation-type biases added little on top, and when used without masking they fell below a vanilla transformer on CWQ (2.1% vs 2.7%). That result constrains our design. On the other side, RGAT Exploiting Typed Syntactic Dependencies for Targeted Sentiment Classification Using Graph Attention Neural Network conditions graph attention on dependency labels for aspect sentiment in reviews, and its ablations show that removing subject and adjectival-modifier relations hurts. That result motivates us. Neither paper tested text in which the literal meaning is inverted, which is exactly where we expect labels to matter.

What Exists and What Is Missing

Three sarcasm models already convolve over dependency graphs, and all three erase edge types. ADGCN Affective Dependency Graph for Sarcasm Detection combines a lexicon-derived "emotional distance" graph with an untyped spaCy parse. It also finds a three-layer depth limit, beyond which word representations blur together. CSDGCN Commonsense Knowledge Enhanced Sentiment Dependency Graph for Sarcasm Detection replaces parse nodes with commonsense-inferred sentiments but keeps a fixed, unlabeled skeleton. IAAD Augmenting Affective Dependency Graph via Iterative Incongruity Graph Learning for Sarcasm Detection learns to down-weight spurious contrast edges, as in "bad memory but great battery life," and explicitly leaves typed edges to future work. We take their parse-based setting but add labels and a target anchor. We also evaluate on hate speech, which none of them do.

On the toxicity side, AmpleHate AmpleHate: Amplifying the Attention for Versatile Implicit Hate Detection locates targets with named-entity tagging and adds a target-to-context attention vector. This is the closest idea to ours. It has no syntax, though, and it breaks when targets are pronouns or informal noun phrases. We find candidate targets from the parse (noun heads and pronouns, flagged if they match an identity lexicon) and let the labeled path, not a flat attention, carry the relation. SharedCon SharedCon: Implicit Hate Speech Detection using Shared Semantics clusters sentence embeddings and uses cluster centers as contrastive anchors. It is the strongest reported system on the Implicit Hate Corpus, but it works at the whole-sentence level. SOC regularization Contextualizing Hate Speech Classifiers with Post-hoc Explanation reduces identity-term bias by penalizing attribution to a fixed word list. We ask whether grammatical role can achieve the same reduction structurally, without the list-based penalty.

Two diagnostic papers motivate focusing on irony specifically. Latent Hatred: A Benchmark for Understanding Implicit Hate Speech shows that adding ConceptNet or Wikidata facts does not help. It argues that implicit hate requires composition rather than lookup, which is an argument for structure and not for knowledge injection. Its taxonomy also isolates an Irony class. How to Solve Few-Shot Abusive Content Detection Using the Data We Actually Have reports that its multi-dataset prompt model still fails on negation and sarcasm. Negation is itself a typed dependency, which makes it a natural probe.

The Module

A pretrained encoder (RoBERTa-base in all runs) produces token vectors. On top of these we stack two or three layers of sparse attention; we stay at three or fewer because ADGCN observed over-smoothing past that depth. For tokens $i$ and $j$, the attention logit is

$$a_{ij} = \frac{q_i^\top k_j}{\sqrt{d}} + b_{\rho(i,j)} + \tau_{ij}\, c_{\rho(i,j)} + \log \pi_{ij}, \quad j \in \mathcal{N}_2(i),$$

and the logit is minus infinity for words outside that neighborhood. The terms are defined as follows. $q_i$ and $k_j$ are the query and key vectors, and $d$ is their dimension. $\mathcal{N}_2(i)$ is the set of words within two arcs of $i$ in the parse. $\rho(i,j)$ is the labeled, directed path signature between the two words, bucketed into a vocabulary of a few hundred signatures such as "amod↑" or "nsubj↓acomp↓." $b_\rho$ is a learned scalar bias per signature. $\tau_{ij}$ is 1 when one endpoint is a candidate target and 0 otherwise. $c_\rho$ is a second bias that applies only on those target-anchored paths. $\pi_{ij}$ is the parser's own probability for the arcs on the path.

The final term brings the edge gating of Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling into attention. Its purpose is to down-weight arcs the parser is unsure about on noisy tweets, which is the failure mode RGAT itself reports. The typed parameters are kept to scalars, and they only act inside the mask. That follows the RASA finding that relation terms help only under masking and hurt when unconstrained. The module's output is fused with the encoder's sentence vector through a learned gate, as in RGAT, and then classified.

Research Program

The work has three parts. The first part builds and replicates. We implement the module and first reproduce RGAT-style numbers on SemEval 2014 restaurants and laptops as a sanity check that our typed attention works where it is known to work. We then train on implicit-hate and sarcasm benchmarks.

The second part isolates the ingredients. Following RASA's methodology, we compare four versions that share the same encoder and parse: the encoder alone, parse mask only, mask plus typed biases, and mask plus typed and target-anchored biases. We add a label-permutation control, in which dependency labels are remapped by a fixed random permutation. The model then keeps its parameter count and connectivity, but the labels lose their meaning. Any gain that survives permutation is extra capacity, not linguistic signal.

The third part probes where the signal lives. We read out the learned $c_\rho$ values and per-example attention mass on target-anchored paths. We also build a small human-verified minimal-pair set, a few hundred sentence pairs taken from Implicit Hate Corpus and NYT-style neutral text. In each pair the same identity term and the same evaluative words appear, but the grammatical role differs: the group as the subject of a mock-compliment versus a group named in an unrelated clause. The prediction is that the typed-target model separates these pairs and the mask-only model does not.

Baselines

The naive reference is the encoder fine-tuned alone with a classification head. Latent Hatred reports 70.4 binary F1 and 58.6 six-way F1 for BERT with augmentation on the Implicit Hate Corpus, and we will rerun this with our RoBERTa backbone rather than rely on that figure. It is the do-nothing point: any structural module that cannot beat a well-tuned plain encoder under the same budget is not worth a parser.

The strongest existing implicit-hate system is SharedCon, which reports 78.5 macro-F1 on the Implicit Hate Corpus and 84.3 on SBIC (SharedCon). We run it with its released recipe on our splits and backbone. Our module is orthogonal to its contrastive loss, so we also run SharedCon plus our module. Beating it means a macro-F1 improvement that holds across five seeds with non-overlapping bootstrap intervals, together with a larger gain on the Irony subclass than on the full set. No per-class Irony number is reported on Lacuna for any system, so we establish that number ourselves.

AmpleHate is the closest mechanism, being target-plus-context attention without syntax. The file gives only an average +6.87 points over contrastive baselines across seven datasets and no per-dataset Implicit Hate Corpus figure, so we rerun it. For sarcasm, the strongest reported numbers come from CSDGCN: 87.16% accuracy on Riloff, 84.95% on Ptáček, and 85.93% on Ghosh (CSDGCN). ADGCN-BERT reports over 90% accuracy on its Twitter-2 split (ADGCN); split definitions may differ, and we will report under both protocols. IAAD claims further gains, but no exact accuracy is reported on Lacuna for Ghosh or Reddit(pol). For identity-term bias, the reference is SOC regularization, which raised accuracy on the NYT neutral set from about 77.8% to 90.1% (Kennedy et al.). Beating it there without an identity word list would be a strong result. Matching it would still be informative.

The key ablations are the mask-only model and the label-permuted model. All systems share the same backbone and the same parser (spaCy transformer model). Each gets an equal hyperparameter search of 20 trials on a shared development split, and the same epoch budget. Results are averaged over five seeds.

Why not just…?

Why not just fine-tune a larger encoder or prompt a large language model? Scale may absorb much of what structure provides, and this is a real threat. No reported result on Lacuna places a large language model on the Implicit Hate Corpus Irony class, and Hangya and Fraser show a prompt-based multi-dataset model still failing on sarcasm and negation. We will include RoBERTa-large fine-tuning and a zero-shot instruction-tuned model as extra references. If either matches our base-sized module on the Irony slice and on the minimal pairs, the structural claim loses its practical case. In that event we would present the probe set as the contribution.

Why not just use the parse as an unlabeled mask, since the knowledge-graph study says labels barely matter? This is the most plausible alternative, and the design is built around it. RASA found masking dominant in knowledge graphs, and ADGCN and IAAD already get strong sarcasm results without labels. Our prediction is narrower than "labels help." It is that labels help on target-anchored inversion and on neutral identity mentions, and not on the corpus average. The mask-only and permuted-label runs settle this directly. If the typed model wins only on average F1 and not on those slices, we will conclude that masking suffices.

Why not just add a target-attention vector or an identity-term penalty, as AmpleHate and SOC do? These address the same two symptoms, missing targets and keyword bias, with no parser at all. They make no use of grammatical role, however. AmpleHate depends on entity tags that miss pronouns and informal groups, and SOC depends on a fixed list of 25 identity words. The minimal-pair set is designed to distinguish role from presence. We will also run SOC on top of our module, to see whether the two effects add or overlap.

Why not just train jointly on sarcasm data without any syntax? The transfer claim, that sarcasm training helps ironic hate, could come from multitask learning alone. We therefore run the plain encoder with the identical sarcasm-plus-hate multitask mixture. The structural claim requires the transfer gain to be larger with typed paths than without.

Proposed Experiments and Decisive Checks

The main evaluation covers the Implicit Hate Corpus (binary, six-way, and the Irony subclass separately). It adds cross-dataset transfer from that corpus to SBIC and DynaHate, the Riloff, Ptáček, and Ghosh sarcasm sets, and the NYT neutral-identity set. We do not use the CONDA gaming chat corpus (Weld et al.) as a main benchmark, because its slang is outside what current parsers handle. Instead we use it as a stress test of the parser-confidence term, reporting implicit-toxicity F1 against its reported 0.59–0.76 range.

Four checks decide the outcome. The first is the transfer test. We train on sarcasm corpora plus the non-irony hate classes, hold out Irony, and ask whether the typed-target model detects held-out ironic hate better than the mask-only and multitask baselines. The second is the permutation test. The typed gain must disappear when labels are permuted. The third is the parse-noise curve. We corrupt a controlled fraction of arcs and labels and check that the confidence term makes accuracy degrade more gently than in RGAT-style hard labels. The fourth is the slice pattern. Gains should concentrate in Irony and Inferiority language rather than in categories like White Grievance, where the signal is plausibly topical.

If all four go the predicted way, typed syntax is shown to be where inversion is encoded. If the permutation or mask-only checks fail, the correct conclusion, consistent with the knowledge-graph evidence, is that connectivity carries the signal in text as well.