Hate speech detectors must be retrained as new coded language, new targets, and new framings appear. Each retraining carries a risk that is easy to miss. The newest data on implicit hate (hostility expressed without slurs, through stereotypes, irony, or dog whistles) is full of identity words like "Muslim," "gay," or "Jew." A model fine-tuned on it can quietly relearn the shortcut "identity word means toxic" and start flagging benign posts about those groups. Ordinary continual learning methods, which keep a model from forgetting old tasks while it learns new ones, protect the wrong thing here. They try to hold old-task accuracy steady. A moderation system needs something else: it must hold its false-positive behavior on benign identity mentions nearly fixed, while staying free to revise what it believes about old hateful content.
We propose an asymmetric stability constraint for updating a toxicity detector. We measure how sensitive the model's predictions are to each parameter on a small, label-free set of benign sentences that mention identity groups. Updates are kept almost flat along those directions. Directions that matter only for previously learned hate are protected softly, so they can be revised. The core claim has two parts. Sequential updates on implicit-hate data bring keyword bias back even in a model that was debiased first. Protecting directions defined by benign identity mentions holds the false-positive rate down better than replaying those same sentences or re-applying a lexicon-based debiasing loss, and this should be most visible on identity groups absent from the anchor set.
Where this sits
The implicit-toxicity literature is entirely static. Contextualizing Hate Speech Classifiers with Post-hoc Explanation adds a penalty on the importance a BERT classifier assigns to 25 fixed identity terms. That penalty raises accuracy on 12,500 benign NYT articles from 77.8% to 90.1% without hurting in-domain F1. Its debiasing objective motivates our measurement. Its fixed lexicon and single offline training run are the limitation we target. Generalizable Implicit Hate Speech Detection Using Contrastive Learning (ImpCon) uses human-written implied statements as contrastive positives to improve cross-dataset F1. Learn from Failure: Causality-guided Contrastive Learning (CCL) mines the errors of a standard classifier as hard positives. Both train once on a fixed dataset. ImpCon's authors report failures on target groups underrepresented in training, which motivates our held-out-group test. Inducing lexicons of in-group language with socio-temporal context and Latent Hatred document that coded language drifts over time, and this is why the detector must be updated at all.
The method comes from continual learning work on safety. The Geometry of Forgetting shows that loss of alignment under fine-tuning closely follows a quadratic form in the Fisher information (a measure of how sharply the model's output distribution reacts to each parameter), with $R^2 = 0.991$. Its FAE method projects updates into the flat directions of that matrix. FAE uses a single Fisher computed on generic preference data for a text generator. We split the Fisher into two pieces with deliberately different strengths, and we apply it to a classifier whose failure mode is false positives.
Unforgotten Safety is the closest warning. Dark Experience Replay (DER) stores old examples with the model's original outputs and replays them. It cut attack success rate on GSM8K from 66.0% to 0.5%, but made the model refuse up to 16.1% of benign requests. This constrains our design: anchoring a moderation model to its past outputs may lock in over-caution instead of preventing it.
Continual Learning with Scaled Gradient Projection supplies the soft-protection mechanism for old hate. Rather than forbidding movement along important old directions, it scales gradients by how important each direction is. SplitLoRA supplies the evidence that one global threshold is a poor choice and that per-layer budgets help. None of these methods has been tested on toxicity or bias objectives.
Core Claim
Let $\theta$ be the detector's parameters and $\delta$ the update in a new training stage. Let $F_B$ be a low-rank-plus-diagonal Fisher of the model's predictive distribution on a benign identity-mention anchor set $B$. It is computed from the model's own outputs, so it needs no labels. Let $F_H$ be the same quantity on a small sample of previously seen hateful data. A stage minimizes
$$\mathcal{L}_{\text{new}}(\theta+\delta) + \frac{\beta}{2}\,\delta^\top F_B\,\delta + \frac{\gamma}{2}\,\delta^\top F_H\,\delta, \qquad \beta \gg \gamma,$$
and is implemented as a preconditioned, scaled projection of each gradient. The two strengths are set differently. $\beta$ is set by a constraint, not by validation accuracy: it is the smallest value that keeps the false-positive rate on a held-out portion of the benign anchors within a fixed tolerance of the pre-update value. $\gamma$ is set by validation F1 on old domains. Following Optimal L2 Regularization in High-dimensional Continual Linear Regression, we also test scaling $\gamma$ as $T/\ln T$ over stages $T$. That result is proven only for linear regression, and here it is treated as a hypothesis.
What would be new is the asymmetry and its source. Stability is defined by benign behavior rather than by old-task loss. Old hateful decisions are deliberately left revisable.
Research Program
The first step is a measurement study, and it determines whether the method is worth building. We start from a debiased detector: HateBERT or RoBERTa with LoRA adapters, trained with the explanation penalty from the SOC paper. We then fine-tune it naively through a stream of implicit-hate data and track false positives on benign identity mentions after every stage.
We use two streams. A dataset stream runs IHC, then SBIC, then ToxiGen, then DynaHate. Their annotation guidelines differ, so it contains genuine label disagreements. A group-incremental stream reveals ToxiGen's 13 target groups a few at a time. Benign anchors are ToxiGen's neutral statements for a subset of groups, plus NYT-style benign identity sentences. The anchors for the remaining groups are held out to test generalization.
If keyword bias does not come back under naive sequential fine-tuning, the gap does not exist. We would report that and stop.
If bias does come back, we build the two-Fisher update with three components. The first is a diagonal Fisher plus a top-$k$ eigenspace estimated by a few gradient passes over the anchors. The second is a per-layer budget in the spirit of SplitLoRA. The third is the constraint-driven tuning of $\beta$. We then analyze which layers and directions carry the identity-sensitive curvature, and whether those directions are shared across identity groups. That sharing is what would let a constraint built from some groups protect others.
Baselines
The naive reference point is sequential LoRA fine-tuning on each stage with no protection. Its job is to quantify how much bias returns.
The upper reference is joint retraining on all data seen so far, with the SOC penalty applied throughout. It is not a fair competitor when old data cannot be kept, but it bounds what any method could reach.
The strongest existing method for keeping bias low is SOC re-applied at every stage. This is the explanation penalty from Contextualizing Hate Speech Classifiers with Post-hoc Explanation, with its 25-term lexicon. On the static task it reports 90.1% NYT accuracy against 77.8% for plain BERT. It is the strongest competitor because it targets exactly the failure we measure.
The strongest existing method against forgetting is DER from Unforgotten Safety. It will be run with a buffer holding both old hateful examples and the benign anchors, plus their stored outputs. Its reported result is 0.5% attack success rate on GSM8K, with up to 16.1% over-refusal. We found no reported result on Lacuna for any continual-learning method on an implicit-toxicity stream, so all continual numbers here will be new. For context only, Latent Hatred reports a static binary F1 of 70.4 on IHC with BERT plus augmentation.
We also run three more baselines: standard EWC with a Fisher computed on old training data, scaled gradient projection on old-task activations, and post-hoc interpolation between the old and new checkpoints with the interpolation weight tuned on validation data.
All methods share several controls. They use the same backbone, LoRA rank, and stream order. They see the same anchor sentences, either as a buffer or as a Fisher source. They get the same number of optimizer steps per stage. Each gets the same 20-trial hyperparameter search, scored by one combined criterion: old-plus-new macro-F1 subject to the false-positive tolerance.
Three ablations isolate the key ingredient:
- Single Fisher: one Fisher on pooled anchors and old hate. This tests whether the asymmetry matters.
- Symmetric strengths: $\beta = \gamma$.
- Accuracy-tuned $\beta$: $\beta$ tuned on validation accuracy instead of by the false-positive constraint.
The method counts as beating the field only if it meets all three of these conditions:
- It matches DER's and SOC's false-positive rate on seen groups within confidence intervals.
- It significantly lowers the false-positive rate on held-out groups.
- Its average F1 over seen domains is no more than one point below the best baseline.
Why not just…?
Why not just replay the benign anchors alongside the new data? This is the most serious threat. Benign identity sentences are cheap to store, and replaying them directly attacks the failure we target. Our argument for curvature is generalization. Replay teaches the model about the specific anchor sentences and groups, while a constraint on identity-sensitive directions could protect groups it never saw. That argument is a hypothesis, and the held-out-group false-positive test decides it. If replay ties on held-out groups, the honest conclusion is that data beats geometry in this setting. We would publish the measurement study and the replay recipe as the result. We also run anchor replay combined with the Fisher penalty, since the two may add up.
Why not just re-apply SOC regularization at every update? SOC penalizes only words on a fixed list. Implicit hate increasingly targets groups and uses coded referents that no list anticipates; ImpCon's authors flag unseen-group failure for this reason. If SOC alone holds false positives on held-out ToxiGen groups as well as our method does, the curvature constraint adds nothing, and the group-incremental stream will show it. SOC also costs extra forward passes on every training step, which we will report alongside the cost of estimating the Fisher.
Why not just retrain jointly on everything? For a model the size of BERT on corpora of tens of thousands of examples, retraining is affordable. It is the right answer whenever all old data can be kept and the old labels are still believed. We include it as the upper bound. The continual setting matters in two cases. One is when old data must be deleted for retention or privacy reasons. The other is when old labels conflict with newer guidelines, as in the IHC-to-SBIC transitions, where joint training averages incompatible labels instead of revising them.
Why not just interpolate between the old and new checkpoints? Interpolation applies one blend weight to every parameter. Towards Better Generalization in Lifelong Person Re-Identification with Flatness-Aware Learning notes that fixed blend weights fail across sequence lengths. A single weight also cannot express the asymmetry we need. Interpolation is nonetheless cheap and strong, so it is a full baseline with a tuned weight.
Decisive Checks
Three results decide the proposal. First, the measurement study must show that naive sequential updates raise the false-positive rate on benign identity mentions above the rate of the debiased starting model. Without that, the work stops. Second, on held-out identity groups, the two-Fisher method must hold false positives significantly lower than anchor replay and per-stage SOC at matched F1. Third, the asymmetry ablation must matter. If a single pooled Fisher or $\beta = \gamma$ performs as well, the contribution reduces to "EWC on a better anchor set." That result would still be useful, but the claim would have to be narrowed accordingly. As a secondary check, we test whether the quadratic prediction from The Geometry of Forgetting also predicts the rise in false-positive rate for a classifier. A strong fit would justify using the benign-anchor Fisher as a cheap early warning before an update ships.
References
- Contextualizing Hate Speech Classifiers with Post-hoc Explanation
- Generalizable Implicit Hate Speech Detection Using Contrastive Learning
- Learn from Failure: Causality-guided Contrastive Learning for Generalizable Implicit Hate Speech Detection
- Inducing lexicons of in-group language with socio-temporal context
- Latent Hatred: A Benchmark for Understanding Implicit Hate Speech
- The Geometry of Forgetting: A Fisher-Information Framework for Alignment-Preserving Continual Adaptation
- Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning
- Continual Learning with Scaled Gradient Projection
- SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting
- Optimal L2 Regularization in High-dimensional Continual Linear Regression
- Towards Better Generalization in Lifelong Person Re-Identification with Flatness-Aware Learning