Paired Benign Perturbation for Evasion-Robust Offensive Language Detection

Offensive-language classifiers are easy to fool. People who want to post abuse replace words with natural-looking disguises, such as "dispicable" for "despicable", and a BERT model trained on the standard OffensEval data loses 32% accuracy on the OLID benchmark and 53% on SOLID (On the Robustness of Offensive Language Classifiers). The obvious fix is to add disguised offensive posts to training. The danger is a pattern familiar from adversarial defenses elsewhere: the defended model loses accuracy on ordinary, unattacked inputs. We argue this cost has a specific cause in offensive-language detection.

If disguised spellings appear in training only on posts labeled offensive, the model learns that nonstandard spelling itself signals offense. Social media is full of harmless nonstandard spelling, so false alarms on benign informal posts should rise. So should confusion at the finer levels of the taxonomy, which ask whether an offense is aimed at someone and at whom. We propose a small change. Apply the same natural-disguise substitutions to non-offensive posts and to every level of the label hierarchy, and keep each post's labels unchanged. This removes the spelling-equals-offense shortcut. The contribution is a diagnosis of where clean accuracy is lost, plus a cheap, testable remedy. We do not propose a new architecture.

Core Claim

The claim has two parts. First, standard evasion-augmented training (retraining on attacked offensive examples) on OffensEval damages clean accuracy in two places that aggregate accuracy hides. One is the false-positive rate on benign posts that carry informal or disguised spellings. The other is the lower, data-poor levels of the label tree.

Second, most of that damage comes from a label-asymmetric shortcut, not from an unavoidable robustness-accuracy trade-off. Paired benign perturbation removes it. It applies the same natural-substitution machinery to non-offensive posts, labeled non-offensive, and to targeted and untargeted posts with their target labels preserved. The expected result is robustness comparable to standard augmentation without the clean-accuracy cost.

If the second part fails, meaning paired perturbation does not remove the cost, that is also informative. It would suggest the trade-off on this task is real rather than a data artifact.

Background and Positioning

OffensEval's dataset, OLID, labels English tweets on three levels (SemEval-2019 Task 6):

  • Level A: offensive or not.
  • Level B: if offensive, a targeted insult or threat, or untargeted profanity.
  • Level C: if targeted, the target is an individual, a group, or other.

Accuracy falls sharply down the tree. The winning BERT system reaches macro-F1 0.8286 on Level A, 0.7159 on B and 0.5598 on C (NULI). SOLID adds millions of silver-labeled tweets and lifts the lower levels (SOLID). These papers supply the task and its fragile lower tiers. They do not study defenses.

The evasion study motivates this work directly (Rusert et al.). It builds word embeddings fine-tuned on deleted tweets that had slipped past moderation, then uses them to generate human-readable disguises; 96% of these keep their offensive meaning. It evaluates only Level A. It mentions adversarial training as future work and never measures clean accuracy under any defense. We reuse its attack and fill that gap.

Two general text defenses constrain the design. FreeLB perturbs continuous word embeddings during training and improves clean scores, for example BERT-base on GLUE from 78.3 to 79.4 (FreeLB). Manifold purification (MC²F) reports that FreeLB-style training holds up poorly against discrete word-substitution attacks. It also claims its own test-time projection method breaks the trade-off, but it was tested only on SST-2, AGNews and Yelp. Those have flat, fairly balanced labels, unlike OffensEval's skewed hierarchy with blurred boundaries.

Work on annotator polarization shows that toxic-language labels are often contested (Polarized Opinion Detection). That makes any method that pushes inputs toward "clean" class regions risky near those boundaries.

Hierarchy-aware adversarial alignment (HiAdv) helps rare classes in topic taxonomies. However, its "adversarial" means a generator and discriminator trained against each other, not defense against attacks. We cite it only as evidence that rare lower-tier classes respond to structural training signals.

Research Program

We train one shared encoder, RoBERTa-base and BERT-base, with three heads, one per OLID level. Levels B and C are trained only on examples where they apply, and inference follows the tree. On top of this model we compare several training regimes.

Disguise lexicon. We take the natural-substitution candidates from the fine-tuned evasion embeddings of Rusert et al., using their released resources if available. The file does not say whether they are public. If they are not, we build a nearest-neighbor misspelling lexicon from SOLID's own tweets and report that as a deviation.

Vocabulary split. We divide the source vocabulary into disjoint train-time and test-time halves, so the model never sees the test-time disguises during training. Without this split, any robustness gain could just be memorization of the attack's word list.

Standard augmentation. Greedy-FT attacks are run against the current model on offensive training posts, and the attacked posts are added with their original labels.

Paired benign perturbation. This adds two things to standard augmentation:

  1. Benign counterparts. The same substitution process is applied to non-offensive posts, labeled non-offensive. We substitute the highest-saliency words and draw from the same disguise distribution, so spelling anomaly occurs at matched rates in both classes.
  2. Tier-consistent pairs. For targeted and untargeted posts, substitutions keep the Level B and C labels. Where a substitution hits a target mention such as a group noun, a small human check confirms the target label still holds. The Level B and C heads are then trained on the pairs as well.

A consistency term asks the model to give the same distribution at every applicable level for a post $x$ and its perturbation $\tilde{x}$:

$$ \mathcal{L} = \sum_{\ell \in \{A,B,C\}} \Big[ \mathrm{CE}(p_\ell(x), y_\ell) + \mathrm{CE}(p_\ell(\tilde{x}), y_\ell) + \lambda\, \mathrm{KL}\big(p_\ell(x)\,\|\,p_\ell(\tilde{x})\big) \Big]. $$

Here $p_\ell$ is the level-$\ell$ head's predicted distribution, $y_\ell$ the gold label at that level (terms are skipped where the level does not apply), CE is cross-entropy, KL is Kullback–Leibler divergence, and $\lambda$ weights the consistency term.

Diagnostic analysis. We measure the spelling-offense shortcut directly as the false-positive rate on held-out non-offensive OLID posts after benign substitution with test-half disguises. We also report per-level clean macro-F1 and per-class recall for untargeted profanity and for the "other" target class. These rare classes are where we predict hidden damage.

Baselines

The naive reference point is the undefended three-head model trained on OLID with NULI-style tweet preprocessing (emoji-to-text substitution, hashtag segmentation, URL normalization, mention truncation). Its clean numbers should be close to the reported NULI results: Level A macro-F1 0.8286 and accuracy 86.28%, Level B 0.7159, Level C 0.5598 (NULI). Its robustness should match the reported 32% accuracy drop under Greedy-FT on OLID (Rusert et al.). Reproducing both anchors confirms our attack and model match prior work.

The strongest direct competitor is standard evasion-augmented training. It is the defense the attack paper itself proposes, and a practitioner would try it first. It uses the same lexicon half, the same number of added examples as our method, the same epochs (1–3, following NULI's report that BERT overfits beyond that) and the same hyperparameter search budget. We found no reported result on Lacuna for adversarial training on OLID or SOLID, so its numbers will be established here.

Two further competitors cover other defense families, and each gets the same encoder and tuning budget:

  • FreeLB is the strongest continuous-embedding method. It is the one most likely to be robust "for free" if the MC²F finding on discrete attacks does not transfer to this task.
  • MC²F (Manifold Purification) represents test-time purification. On AGNews under BERT-Attack it reports adversarial accuracy of 45.3% against the strongest baseline's 38.6%, with clean accuracy retained. If we cannot reimplement its Riemannian flow faithfully, we will report that and use a simpler purification: a normalizing flow over sentence embeddings with gradient projection toward high-likelihood regions.

Two ablations isolate the key ingredient:

  • Benign pairs only at Level A. This tests whether tier-consistent pairs matter beyond breaking the Level A shortcut.
  • Benign random-typo pairs in place of natural disguises. This tests whether the benign perturbations must match the attack's distribution.

What counts as winning. Paired benign perturbation beats standard augmentation if it:

  • matches its robust accuracy within one standard deviation over five seeds,
  • recovers at least half of its clean Level B and C macro-F1 loss relative to the undefended model, and
  • cuts its benign-perturbed false-positive rate significantly (paired bootstrap over test posts).

Why not just…?

Why not just normalize spelling before the classifier? Rusert et al. report that simple typos and inserted characters are easily blocked by such shielding. Their attack uses real words mined from evasive tweets, and a spell-checker would leave those alone or correct them to the wrong word. The decipherment study found noisy-channel spelling correction reached only 65% accuracy under leetspeak (Decipherment). We still run a normalizer front-end on both clean and attacked test sets, because rewriting benign slang may itself cost clean accuracy. If it matches our robustness at no clean cost, the proposal is unnecessary.

Why not just use FreeLB? It is the most serious threat. Its clean gains on GLUE are real. MC²F's finding that it is weak against discrete attacks comes from sentiment and topic tasks, not tweets. The FreeLB baseline above settles the question directly. We also test FreeLB combined with paired perturbation, in case the two are complementary rather than competing.

Why not just train on SOLID's millions of tweets? Rusert et al. found the SOLID-trained model more fragile (a 53% drop) than the OLID-trained one (32%). Scale alone therefore does not fix evasion. We still include an OLID-plus-SOLID-silver model as a reference, because SOLID's gains at Levels B and C could by themselves reduce the lower-tier clean loss we attribute to the shortcut.

Why not filter by perplexity or paraphrase with a language model? Perplexity filtering flagged about 10% of benign, well-formed prompts in Baseline Defenses. Informal tweets should score worse still. Paraphrasing risks erasing exactly the cue that separates venting from a targeted insult. We measure the perplexity filter's rejection rate on non-offensive OLID and SOLID test posts to quantify this, rather than assert it.

Proposed Experiments

Data. Models are trained on the OLID training split (13,240 tweets). Clean evaluation uses the 860-tweet OLID test set per level, plus the SOLID test set split into its Easy and Hard halves. Hard tweets, where a curse-word baseline fails despite 0.93 F1 on Easy, test whether defenses lose subtle signal.

Attacks. Robustness is measured under:

  • Greedy-FT and attention-based FT substitution using the held-out lexicon half, extended to all three levels,
  • TextFooler and BERT-Attack, for cross-attack transfer, and
  • an adaptive Greedy-FT run against each defended model.

First decisive check: does the shortcut exist? It exists if standard augmentation raises the benign-perturbed false-positive rate well above the undefended model's while leaving the unperturbed Level A score nearly flat. If it does not, the diagnosis is wrong and the paper becomes a negative result on defenses for this task.

Second decisive check: does pairing fix it? Paired perturbation should restore the false-positive rate and the recall of untargeted profanity and the "other" target class without giving up robust accuracy. The random-typo ablation should fail to do the same. If the random-typo ablation works equally well, the mechanism is generic anomaly exposure, not distribution matching, and we will report it that way.

Third check: annotator disagreement. On SOLID, which records how strongly its labeling ensemble disagreed on each tweet, we test whether defense-induced errors concentrate on high-disagreement tweets. That would link clean degradation to the blurred boundaries that polarization work describes.