Detoxifying LLMs via Representation Erasure-Based Preference Optimization

Nazanin Mohammadi Sepahvand, Eleni Triantafillou, Hugo Larochelle, Doina Precup, Daniel M. Roy, Gintare Karolina DziugaitearXiv 2026 · arXiv

TL;DR

Current detoxification methods for LLMs (like DPO or NPO) act as superficial "output suppressors" that can be easily bypassed by jailbreaks or reversed via lightweight fine-tuning. This paper introduces REPO (Representation Erasure-based Preference Optimization), a method that scrubs toxic features directly from the model's internal representations at a token-granular level. REPO achieves state-of-the-art robustness against adaptive jailbreaks and relearning attacks while perfectly preserving general model utility.

What Problem Does This Paper Address?

When we "align" or detoxify large language models using standard preference optimization algorithms like Direct Preference Optimization (DPO) or Negative Preference Optimization (NPO), we are mostly just shifting output token probabilities. Mechanistic analyses show that this safety training is skin-deep: the underlying features and "toxic directions" remain completely intact within the model's hidden representations.

Because the toxic knowledge is never actually removed, these defenses are fragile. Adversaries can easily exploit them using white-box jailbreak methods like Greedy Coordinate Gradient (GCG), or completely undo the safety training via "relearning attacks"—fine-tuning the model on as few as ten harmful examples. While some representation-level unlearning methods exist (like RMU or Circuit Breakers), they often degrade the model's general capabilities or fail against highly adaptive attacks because they lack precision.

Methods

REPO reformulates LLM detoxification as a token-level preference problem enforced directly in the representation space rather than the output space.

Given a prompt, REPO utilizes pairwise supervision: a preferred retain continuation ($x_r$, non-toxic) and a dispreferred forget continuation ($x_f$, toxic). The training architecture optimizes two core objectives simultaneously:

  1. Token-Level Retain Anchoring Loss: A token-wise KL divergence loss that anchors the edited model to a frozen reference model on benign text. This ensures the model doesn't lose its general language capabilities.
  2. Token-Granular Adversarial Erasure Loss: A small discriminator module (a two-layer MLP) is attached to a deep transformer layer (just before the unembedding layer). It is trained to predict whether a token representation originated from a retain or a forget sequence. Crucially, the LLM is trained to fool this discriminator using a Gradient Reversal Layer (GRL). This forces the internal representations of toxic continuations to mathematically converge toward their benign counterparts, effectively erasing the distinguishing toxic features.
Figure 1: A schematic representation of REPO. Its regressor can be attached to any transformer block M targeted for unlearning; here, M is taken as the final transformer block before the linear unembedding layer. For each prompt, the retain (nontoxic) continuation xr and the forget (toxic) continuation xf are fed into the network, and the discriminator is trained to distinguish between toxic and nontoxic inputs.

Datasets

The authors evaluated REPO using several standard and pairwise datasets:

  • PairToxicity: A pairwise dataset consisting of 24,576 prompt-continuation pairs. The non-toxic continuations (retain set) were generated via greedy decoding, and the toxic continuations (forget set) were generated using Plug and Play Language Models (PPLM) guided by a toxicity probe.
  • WikiText-2: Used to evaluate general language modeling utility via perplexity and $F_1$ scores.
  • RealToxicityPrompts: A challenge set of 1,199 prompts used to assess out-of-distribution (OOD) toxicity mitigation.
  • MMLU (Massive Multitask Language Understanding): Benchmarked on Gemma-2B to ensure downstream task performance was not degraded.

Key Insights and Ideas

  • Output Space vs. Representation Space: Shifting probabilities at the output layer leaves the internal "affordances" for toxicity intact. True robustness requires representation erasure, making downstream computation blind to toxic directions.
  • The Power of Token-Level Granularity: Sequence-level or segment-averaged adversarial training causes localization to vanish, resulting in diffuse, messy edits across the network. Applying the adversarial discriminator per token allows REPO to make highly localized, surgical edits to the exact computation graphs and neurons encoding toxicity without disturbing adjacent tokens.
  • Deep, Targeted Interventions: Mechanistic analysis revealed that REPO concentrates its weight modifications in the deeper layers of the network. Specifically, it targets and suppresses activations in the top neurons most heavily aligned with the toxic direction ($W_{toxic}$), leaving neutrally aligned neurons untouched.

Key Results

REPO was tested across multiple model families (GPT-2 Small, GPT-2 Medium, and Gemma-2B) and consistently outperformed output-based and representation-based baselines.

  • Detoxification Performance: On GPT-2 Small, REPO achieved an in-distribution forget toxicity score of 0.1020 (compared to DPO's 0.1506 and RMU's 0.1527) while maintaining a baseline-matching WikiText perplexity of 28.23.
  • Out-of-Distribution Generalization: On RealToxicityPrompts, REPO achieved the lowest toxicity score (0.1913) while preserving language fluency.
Figure 2: Detoxified models vs reference. (Left) Perplexity vs. toxicity ratios on PairToxicity (in-distribution); (Middle) Perplexity vs. toxicity ratios on WikiText/RealToxicity (OOD); (Right) F1 ratio on WikiText vs. OOD toxicity. Each point is a model–method pair. The green gradient highlights lower toxicity and ratios near 1, darkest at the ideal point (x=1, y=0). Dashed gray lines mark ratio = 1 for easy comparison to the reference.
  • Resisting Relearning Attacks: When subjected to fine-tuning attacks designed to make the model re-learn toxic behavior, REPO maintained a massive safety margin over DPO and NPO, showing significantly lower post-attack toxicity across varying fine-tuning subset sizes.
Figure 3: Average toxicity after the Relearning Attack for different subset sizes across methods on GPT2-small. (Top) OOD toxicity (RealToxicity); (Bottom) In-distribution toxicity (pairwise set). Dashed horizontal lines indicate each method’s baseline toxicity before the attack.
  • Downstream Utility: On the MMLU benchmark using Gemma-2B, REPO achieved an accuracy of 0.422, completely preserving (and slightly exceeding) the unaligned reference model's accuracy of 0.418.

Limitations and Future Work

  • Model Scale: To allow for exhaustive, token-level mechanistic analysis (such as layer-token drift and weight-space distances), the authors restricted their evaluations to smaller open models (up to Gemma-2B). Validating REPO on 13B+ or MoE models like Mixtral remains an open direction.
  • Scope of Unlearning: The paper focuses entirely on toxicity as a case study. The authors note that the framework is generalizable, but future work is needed to test REPO on other critical unlearning tasks, such as removing copyright material, private personal information (PII), or hazardous cyber/biological knowledge.

Summary updated

Abstract

Large language models (LLMs) trained on webscale data can produce toxic outputs, raising concerns for safe deployment. Prior defenses, based on applications of DPO, NPO, and similar algorithms, reduce the likelihood of harmful continuations, but not robustly so: they are vulnerable to adversarial prompting and easily undone by fine-tuning-based relearning attacks. Indeed, research has shown that these edits to the model are superficial: linear probing reveals that harmful "directions" remain present in representations. To address this, we propose Representation Erasure-based Preference Optimization (REPO), reformulating detoxification as a token-level preference problem. Using a novel objective with preference data, we force the representations of toxic continuations to converge toward their benign counterparts. Our mechanistic analysis reveals that this granular approach is critical: unlike baselines, REPO induces deep, localized edits to toxicity-encoding neurons while preserving general model utility. Exhaustive evaluations show that REPO achieves state-of-the-art robustness, stopping sophisticated threats-including relearning attacks and enhanced GCG jailbreaks-where existing representation- and output-based methods fail.
BibTeX
@misc{sepahvand2026detoxifying,
  author = {Nazanin Mohammadi Sepahvand and Eleni Triantafillou and Hugo Larochelle and Doina Precup and Daniel M. Roy and Gintare Karolina Dziugaite},
  title = {{Detoxifying LLMs via Representation Erasure-Based Preference Optimization}},
  year = {2026},
  eprint = {2602.23391},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.23391},
}

Loading related content...

Figure