Implicit hate speech is abuse that avoids slurs and profanity and relies on stereotypes, irony, or coded phrasing. It is expensive to annotate, so the field increasingly trains detectors on text written by large language models (LLMs). This works up to a point. Fine-tuning on the machine-generated ToxiGen corpus improves detection on human-written benchmarks by 7–19% (ToxiGen). Yet other work reports that standard BERT classifiers struggle to learn transferable features from Llama-3 paraphrases (Towards Generalizable Generic Harmful Speech Datasets). Computer vision has a precise account of this kind of friction. Images from a frozen generator share a small set of "this is synthetic" feature directions, and a classifier can exploit those directions instead of the real class content (DREAM). Nobody has checked whether LLM-written hate speech carries the same kind of shortcut, which would be a stylistic fingerprint that a detector can mistake for toxicity.
We propose to measure that shortcut in text and then break it during training. Each post, synthetic or real, is tied to a structured frame: a small record of who is targeted, which rhetorical device is used, and what the post implies. Real and synthetic posts that share a frame become positive pairs in a contrastive objective, meaning a loss that pulls matched pairs together in embedding space and pushes mismatched ones apart. Because every frame is represented by both kinds of text, "was this written by an LLM?" carries no information about which frame a post matches. The encoder is therefore pushed to discard the generator fingerprint rather than learn around it. In two sentences: LLM-generated hate data probably teaches detectors the generator's style. We remove that incentive by aligning every post, synthetic or real, to a shared structured description of its meaning, with each description represented by both domains.
Core Claim
The claim has two parts, and the second only makes sense if the first holds. The first is empirical. Encoders fine-tuned on a mix of real and LLM-generated implicit hate data contain a low-dimensional subspace that separates synthetic from real text. When synthetic data is unevenly distributed across labels or target groups, that subspace becomes predictive of the toxicity label. The second is methodological. Contrastive alignment between surface text and structured frames, with batches built so that each frame appears in both domains, removes this subspace's usefulness during training. We expect this to transfer to human-written test sets better than either naive mixing or applying DREAM-style subspace projection after training.
Each prior line of work leaves part of this problem open. ImpCon aligns a post with its human-written implied statement and gains up to 9.10% in cross-dataset transfer on BERT. That method provides our alignment idea, but it uses only real posts and never confronts synthetic data. SharedCon replaces human implications with cluster centroids drawn from existing posts. Its own design implies that once synthetic posts are added, clusters could form around domain identity rather than meaning. GraphCLIP and BALI show that aligning text with structured representations transfers well, which motivates treating frames as a second modality. Both work on fact-like structure, however, and neither studies a synthetic-versus-real gap. DREAM provides the diagnosis and a post-hoc fix, but only for images. Its own stated open problem is that pooling several generators may make the shortcut spread out and non-linear. A training-time fix that never assumes a single subspace is meant to address exactly that case.
Research Program
Measuring the shortcut. The first stage asks whether the shortcut is real. We fine-tune HateBERT on real data alone and on real data mixed with synthetic posts. For each model we collect sentence features and fit a linear probe that predicts synthetic versus real. Following DREAM, we then compute the top singular directions of the difference between synthetic and real feature covariances. Two quantities matter. One is how much toxicity-label information the probe recovers when restricted to those few directions. The other is how much held-out real performance falls when they are projected out.
We also run a rewrite test. We take real benign and real toxic test posts and have the generator LLM rewrite each one while keeping its meaning. We then measure how often the detector's prediction flips. If the detector relies on generator style, LLM-rewritten benign posts will be flagged more often than their originals. If this stage finds no structured shortcut, the project changes. It becomes a negative result about text generators, and the method stage is dropped.
Building synthetic data from frames. Frames come from existing annotations. Social Bias Frames supplies target group, intent, offensiveness, and free-text implied statement for posts in the Social Bias Inference Corpus (SBIC). The Implicit Hate Corpus (IHC) from Latent Hatred adds targets, implied statements, and a six-way rhetorical taxonomy covering categories such as irony, stereotype, and white grievance. Benign posts that mention a group get a frame stating that target with no implication.
For each training frame, an open LLM writes several new posts conditioned on the frame. ToxiGen statements are mapped to frames by target group and label. Conditioning on frames from real annotations is meant to limit the "generic stereotype" collapse that Social Bias Frames documents for free generation, because the implication is fixed in advance. A random sample of generated posts will be checked by humans for label fidelity.
The training objective. A text encoder $g$ embeds posts and a frame encoder $h$ embeds frames. For a post $x$ with frame $f$, the alignment loss is
$$\mathcal{L}_{\text{align}} = -\log \frac{\exp(\langle g(x), h(f)\rangle/\tau)}{\sum_{f' \in B} \exp(\langle g(x), h(f')\rangle/\tau)},$$
where $B$ is the set of frames in the batch, $\langle\cdot,\cdot\rangle$ is cosine similarity, and $\tau$ is a temperature. The key design choice is batch construction. Every frame in $B$ is represented by at least one real and one synthetic post, and negative frames are drawn equally from both domains. Under that sampling, domain identity is independent of which frame is the positive, so no direction encoding domain can lower the loss.
Frame embeddings are computed only from their structured fields, never from synthetic text. In DREAM's terms they act as real-anchored prototypes. The total loss adds standard cross-entropy on real labels, plus cross-entropy on synthetic labels at a tuned weight.
The frame encoder is either a sentence encoder over the frame written out as text, or a small graph attention network over typed nodes for target, rhetorical device, and implied statement. Which one is better is left as an ablation, not an assumption. BALI found that large text backbones handle flattened graph text competitively, so the claim does not rest on the graph encoder.
Baselines
The naive reference point is HateBERT fine-tuned on real training data only, with cross-entropy. The second reference is the same model trained on real data mixed with synthetic data. This is the ToxiGen recipe, which ToxiGen reports gives 7–19% gains on human benchmarks. It is the method we must beat on transfer, and it is the condition in which the shortcut should appear. We also report BERT with augmentation from Latent Hatred, at 70.4 binary F1 on IHC, as an anchor for IHC binary results.
The strongest existing detector is SharedCon. It reports 78.5 macro-F1 on IHC, 84.3 macro-F1 on SBIC, and a 4.5-point cross-dataset gain from IHC to SBIC. We rerun it twice: on real data alone, and with the same synthetic pool added. Beating it means matching it in-domain, within one macro-F1 point, while exceeding it on cross-dataset transfer and on the rewrite-flip test.
ImpCon, which reports up to a 9.10% cross-dataset gain on BERT, is run in three forms. The first uses human implications and serves as a text-only alignment reference. The second uses LLM-generated implications. The third adds synthetic posts aligned to their implications without enforcing that each frame appears in both domains. That third variant is the ablation that isolates our key ingredient. AmpleHate reports a 6.87-point average gain over contrastive baselines across seven datasets. No per-dataset reported result was found on Lacuna for AmpleHate on IHC or SBIC, so we rerun it.
The DREAM-style text baseline applies domain centering and soft SVD projection to the mixed-data model's features after training. A domain-adversarial baseline trains a synthetic-versus-real discriminator with a reversed gradient. For DREAM itself the only reported numbers are in vision, at 69.73% task-agnostic accuracy on CIFAR-100 versus 62.04% for SEED, so its text form gets no target number and is tuned by us.
All methods share the same backbone, the same real and synthetic pools, and the same number of training steps. Each method gets an equal hyperparameter search of twenty configurations, selected on real validation data only. Results are averaged over five seeds and reported with confidence intervals.
Why not just…?
Why not just mix in the synthetic data and fine-tune? ToxiGen shows this already helps, so it is a real competitor. Almohaimeed et al. report friction when BERT learns from Llama-3 paraphrases, but they do not explain it. If the measurement stage finds no predictive domain subspace and naive mixing matches our method on transfer and rewrite flips, that result is the finding and we report it.
Why not just project the domain subspace out afterward, as DREAM does? This is the most serious threat, because it is cheap and needs no frames. Our argument rests on DREAM's own stated limitation. Hard projection removes useful features, and pooled generators spread the shortcut across many directions. We test this directly by training on synthetic data from two or three different LLMs. If post-hoc projection matches frame-anchored training in that setting, the frame machinery is unnecessary.
Why not just use ImpCon with LLM-written implications? That keeps alignment and drops annotation cost, and it is close to what we do. The difference is the domain-balanced pairing. The ablation that keeps everything except that constraint answers whether the pairing adds anything.
Why not just prompt a large LLM to classify? Cracking the Code found embedding-based methods beat prompting and that rich prompts cause overcorrection. Almohaimeed et al. report that GPT-4o still misreads implicit hate. We include a zero-shot LLM row as a reference, but deployed moderation needs a small encoder in any case.
Why not plug in Wikidata or ConceptNet? Latent Hatred tested exactly this and found no significant gain. That negative result is why our structure encodes rhetoric and implication rather than entity facts.
Proposed Experiments
Training and evaluation use IHC, SBIC, DynaHate, and ToxiGen's human-validated subset. The headline metric is cross-dataset macro-F1: train on IHC and test on SBIC and DynaHate, and the reverse. Shortcut metrics come from the measurement stage: probe accuracy for synthetic versus real, label information in the top domain directions, and the prediction-flip rate under LLM rewriting. Rewrites are produced both by the training generator and by an unseen LLM, which tests whether the fingerprint transfers across generators.
Three outcomes would decide the project. First, naive mixing must show a measurable shortcut, meaning label information concentrated in a few domain directions and elevated flip rates on rewritten benign posts. Without that, the premise fails. Second, frame-anchored training must cut flip rates and probe-recoverable label information relative to naive mixing and to the unbalanced-pairing ablation, while matching SharedCon in-domain. Third, its cross-dataset macro-F1 must exceed both SharedCon and post-hoc projection in the multi-generator setting.
A stress test makes the shortcut deliberately easy to exploit by skewing synthetic data toward toxic labels for half of the target groups. A method that truly ignores generator style should lose little on real test posts from those groups. A secondary analysis compares frame encoders, the graph network against the written-out text version. It is reported as is, with no prior commitment to the graph.
References
- ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
- Towards Generalizable Generic Harmful Speech Datasets for Implicit Hate Speech Detection
- Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning
- Generalizable Implicit Hate Speech Detection Using Contrastive Learning
- SharedCon: Implicit Hate Speech Detection using Shared Semantics
- GraphCLIP: Enhancing Transferability in Graph Foundation Models for Text-Attributed Graphs
- BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignment
- Social Bias Frames: Reasoning about Social and Power Implications of Language
- Latent Hatred: A Benchmark for Understanding Implicit Hate Speech
- AmpleHate: Amplifying the Attention for Versatile Implicit Hate Detection
- Cracking the Code: Enhancing Implicit Hate Speech Detection through Coding Classification