Subtracting the Target: Offset Representations for Implicit Hate Against Unseen Groups

Implicit hate speech carries its harm without slurs. "We shouldn't lower our standards to hire more women" contains no abusive word, yet it asserts that women are less qualified. Detectors trained on such posts fail in a specific, documented way: they get worse when the post targets a group that was rare or absent in training. A large share of implicit hate is the same rhetorical move (calling a group lazy, dangerous, or undeserving) applied to different groups. That suggests a representation that splits each post into who is targeted and what is being said about them. The second part could then carry over to groups the detector has never seen.

We propose to build that split with an old idea from word embeddings: vector offsets. In word vector spaces, the difference between two vectors (king minus man) often encodes a relation that transfers to new pairs (woman → queen). We represent each post by the offset between its sentence embedding and the embedding of its target group, $r(p) = z(p) - z(g)$. We train the encoder so that posts making the same implied claim about different groups produce parallel offsets. The classifier reads only the offset, and target identity is kept as a separate channel. The claim to test is narrow: if implicit stereotyping is close to "group plus predicate" in a suitably trained space, then subtracting the target gives better detection on held-out target groups than collapsing posts onto their implications or their cluster centroids.

Core Claim

Current representation-learning methods for implicit hate pull each post toward a point. ImpCon (Generalizable Implicit Hate Speech Detection Using Contrastive Learning) pulls a post toward its annotated free-text implication. SharedCon (SharedCon: Implicit Hate Speech Detection using Shared Semantics) pulls posts toward a K-means centroid post. Neither separates the target from the claim, and ImpCon's authors report that their model still struggles with groups underrepresented in training. We claim that a relational objective can make the stereotype predicate a direction that is shared across groups. Such an objective would be applied to target-subtracted offsets and would make offsets parallel when implications share a predicate. This directly attacks the unseen-target failure.

The offset is non-commutative (post minus target is not target minus post), which matters because the relation is directed. As a side effect, the offset should reduce false positives driven by identity words alone, because the group component is removed before classification rather than penalized.

Research Program

The method borrows its main ingredient from the analogy literature. Can language models learn analogical reasoning? trains a Siamese BERT ("BERT a-b") to maximize cosine similarity between difference vectors of analogous pairs. With a few thousand examples it raised Distractor-set accuracy from 0.53 to 0.69 without damaging word similarity. Embedding Semantic Relations into Word Representations similarly treats offsets as an explicit training target. Compositional Approaches for Representing Relations Between Words found subtraction (PairDiff) to be the best operator for relation classification, at 87.38% on DiffVecs. All of these work on word pairs. We move the objective to sentence-level posts and their target groups.

Building analogy pairs. We construct the training signal from the implication annotations in the Social Bias Inference Corpus (SBIC) and the Implicit Hate Corpus (IHC). The SBIC implications come from Social Bias Frames and the IHC implications from Latent Hatred. Implications mostly take the form "[group] [predicate]". We strip the group mention and embed the remaining predicate. Posts are grouped into predicate clusters using a frozen sentence encoder, in the same spirit as SharedCon's clustering. Two posts about different groups whose implications fall in the same predicate cluster form an analogy pair.

Training objective. The encoder $f$ embeds posts. The same encoder embeds group names to give $z(g)$, and a learned linear map $W$ lets the subtraction act in a projected space. The objective is

$$\mathcal{L} = \mathcal{L}_{\text{CE}}\big(h(r(p))\big) + \lambda\,\mathcal{L}_{\text{par}}, \qquad r(p) = W\big(f(p) - f(g_p)\big),$$

where $h$ is the classifier head and $g_p$ is the post's target group. $\mathcal{L}_{\text{par}}$ is a contrastive loss that treats the offsets of an analogy pair as positives and the offsets of same-group, different-predicate posts as negatives. The negatives are the important part. They force the offset to encode the claim rather than the group, which is the failure that The (too Many) Problems of Analogical Reasoning with Word Vectors warns about: offsets that stay near their starting point, the "proximity trap."

Non-hateful posts and inference. Non-hateful posts that mention a group still get an offset. Their offsets should carry no stereotype direction.

At test time, the target group comes from one of two sources. We report both: gold target annotations, and targets predicted by a small target generator trained as in Social Bias Frames, which reports target-group BLEU in the 70s–80s. Reporting both separates the representation's value from errors in finding the target.

Diagnostic geometry. Separately from accuracy, we check whether the offsets carry the structure we intend. A linear probe should be unable to recover the target group from $r(p)$ (near chance), while hate and implicit-category labels should remain decodable from it. We also run an "honest" analogy retrieval in Rogers et al.'s sense, without excluding source items. For a post about group X, we form $f(p) - f(X) + f(Y)$ and check whether its nearest neighbors are posts about Y that share the predicate, rather than the source post itself.

Baselines

The naive reference point is a BERT or HateBERT classifier fine-tuned with cross-entropy, trained and evaluated on our held-out-target splits. In-domain, Latent Hatred reports 70.4 binary F1 for BERT with augmentation on IHC. ImpCon reports that cross-entropy baselines drop more than 12.5 F1 across datasets.

The strongest existing detector is SharedCon. It reports 78.5 macro-F1 on IHC and 84.3 macro-F1 on SBIC, plus a 4.5-point cross-dataset gain over the previous best when training on IHC and testing on SBIC. ImpCon is the closest method in spirit, since it also consumes implication annotations; it reports cross-dataset gains of up to 9.10% on BERT and 8.71% on HateBERT.

Both will be rerun with the same backbones (BERT-base and HateBERT), the same training data, the same number of epochs, and the same hyperparameter search budget. ImpCon will receive the same implication annotations our method uses. There is no reported result found on Lacuna for any method on a held-out-target-group split, so all numbers on that split will be produced here.

We also include the frozen-LLM embedding approach of Cracking the Code, which reports 0.6121 F1 on Latent-hatred, as a reference for frozen-feature methods. For identity-word false positives, we use SOC regularization from Contextualizing Hate Speech Classifiers with Post-hoc Explanation, which lifts NYT adversarial accuracy from about 77.8% to 90.1%.

Four ablations isolate the key ingredient. The first concatenates $f(p)$ and $f(g)$ instead of subtracting them. The second subtracts without the parallel-offset loss. The third applies the parallel loss with the negatives removed. The fourth replaces the predicate-based analogy pairs with random cross-group pairs; this is the analogue of the random-tuple control in Tox-BART.

Beating SharedCon means a statistically significant macro-F1 gain on held-out target groups, measured over seeds, without losing more than about one point in-domain. Beating the ablations means the subtraction and the parallel loss each contribute on that split.

Why not just…?

Why not just augment with counterfactual identity swaps? Replacing group words with other groups' names and requiring the same prediction is the cheapest route to target invariance, and it is the most serious threat. It requires a surface token to swap. Much implicit hate refers to its target only indirectly, through pronouns, dog whistles such as #WPWW or ((( ))), or tropes (see Latent Hatred). Our method instead subtracts an annotated or predicted target, which still works when the group is never named. We will run counterfactual swapping as a full baseline and report results separately on posts with and without an explicit group mention. If swapping matches us on both subsets, the offset adds nothing, and we will say so.

Why not just use ImpCon or SharedCon with more target-diverse data? These methods make posts with the same meaning look alike, but "same meaning" includes the target. Two posts calling different groups lazy are therefore not pushed together, and ImpCon's authors themselves name underrepresented targets as the remaining failure. The held-out-target split tests exactly this. If SharedCon's centroids already transfer as well as our offsets, the geometric hypothesis fails.

Why not just prompt a large model? Prompting is weak on the closest evidence available. Cracking the Code finds that frozen-embedding classifiers beat prompting by roughly 5 F1 on average. Tox-BART finds that GPT-3.5 gives vague, guardrail-shaped readings of implicit hate. We include a few-shot prompted open model as a reference rather than as the main competitor.

Why not just mask identity words? Kennedy et al. show that removing identity terms destroys the model's knowledge of who is targeted and hurts detection. Our design keeps the target as a separate channel instead of deleting it.

Proposed Experiments

Main experiment: held-out target groups. We normalize SBIC's 1,414 target groups and IHC's target annotations into canonical groups. We then hold out a set of mid- and low-frequency groups entirely from training and evaluate macro-F1 on posts that target them, averaged over five group partitions and three seeds.

Secondary experiments. We run cross-dataset transfer from IHC to SBIC and back, so results can be compared with the ImpCon and SharedCon protocols. We test on DynaHate, which has no implication annotations, to check that the method does not depend on them at inference. We measure false-positive rate on the NYT adversarial set.

Decisive checks. Three outcomes would falsify the claim. The first is that, on held-out groups, the full model fails to beat both SharedCon and counterfactual swapping. The second is that the random-pair ablation matches the predicate-pair model, which would mean the gain comes from regularization rather than relational structure (the same trap Tox-BART exposed for knowledge graphs). The third is that the target probe can still decode group identity from $r(p)$, or that honest analogy retrieval returns the source post, which would mean the subtraction has fallen into the proximity trap.

A positive result needs all three: a held-out-group gain, a drop in target decodability, and above-chance cross-group analogy retrieval. Together they would show that a large part of implicit stereotyping is shared structure across groups that a detector can learn once and reuse.