TL;DR
Current detoxification methods for LLMs (like DPO or NPO) act as superficial "output suppressors" that can be easily bypassed by jailbreaks or reversed via lightweight fine-tuning. This paper introduces REPO (Representation Erasure-based Preference Optimization), a method that scrubs toxic features directly from the model's internal representations at a token-granular level. REPO achieves state-of-the-art robustness against adaptive jailbreaks and relearning attacks while perfectly preserving general model utility.
What Problem Does This Paper Address?
When we "align" or detoxify large language models using standard preference optimization algorithms like Direct Preference Optimization (DPO) or Negative Preference Optimization (NPO), we are mostly just shifting output token probabilities. Mechanistic analyses show that this safety training is skin-deep: the underlying features and "toxic directions" remain completely intact within the model's hidden representations.
Because the toxic knowledge is never actually removed, these defenses are fragile. Adversaries can easily exploit them using white-box jailbreak methods like Greedy Coordinate Gradient (GCG), or completely undo the safety training via "relearning attacks"—fine-tuning the model on as few as ten harmful examples. While some representation-level unlearning methods exist (like RMU or Circuit Breakers), they often degrade the model's general capabilities or fail against highly adaptive attacks because they lack precision.
Methods
REPO reformulates LLM detoxification as a token-level preference problem enforced directly in the representation space rather than the output space.
Given a prompt, REPO utilizes pairwise supervision: a preferred retain continuation ($x_r$, non-toxic) and a dispreferred forget continuation ($x_f$, toxic). The training architecture optimizes two core objectives simultaneously:
- Token-Level Retain Anchoring Loss: A token-wise KL divergence loss that anchors the edited model to a frozen reference model on benign text. This ensures the model doesn't lose its general language capabilities.
- Token-Granular Adversarial Erasure Loss: A small discriminator module (a two-layer MLP) is attached to a deep transformer layer (just before the unembedding layer). It is trained to predict whether a token representation originated from a retain or a forget sequence. Crucially, the LLM is trained to fool this discriminator using a Gradient Reversal Layer (GRL). This forces the internal representations of toxic continuations to mathematically converge toward their benign counterparts, effectively erasing the distinguishing toxic features.
Datasets
The authors evaluated REPO using several standard and pairwise datasets:
- PairToxicity: A pairwise dataset consisting of 24,576 prompt-continuation pairs. The non-toxic continuations (retain set) were generated via greedy decoding, and the toxic continuations (forget set) were generated using Plug and Play Language Models (PPLM) guided by a toxicity probe.
- WikiText-2: Used to evaluate general language modeling utility via perplexity and $F_1$ scores.
- RealToxicityPrompts: A challenge set of 1,199 prompts used to assess out-of-distribution (OOD) toxicity mitigation.
- MMLU (Massive Multitask Language Understanding): Benchmarked on Gemma-2B to ensure downstream task performance was not degraded.
Key Insights and Ideas
- Output Space vs. Representation Space: Shifting probabilities at the output layer leaves the internal "affordances" for toxicity intact. True robustness requires representation erasure, making downstream computation blind to toxic directions.
- The Power of Token-Level Granularity: Sequence-level or segment-averaged adversarial training causes localization to vanish, resulting in diffuse, messy edits across the network. Applying the adversarial discriminator per token allows REPO to make highly localized, surgical edits to the exact computation graphs and neurons encoding toxicity without disturbing adjacent tokens.
- Deep, Targeted Interventions: Mechanistic analysis revealed that REPO concentrates its weight modifications in the deeper layers of the network. Specifically, it targets and suppresses activations in the top neurons most heavily aligned with the toxic direction ($W_{toxic}$), leaving neutrally aligned neurons untouched.
Key Results
REPO was tested across multiple model families (GPT-2 Small, GPT-2 Medium, and Gemma-2B) and consistently outperformed output-based and representation-based baselines.
- Detoxification Performance: On GPT-2 Small, REPO achieved an in-distribution forget toxicity score of 0.1020 (compared to DPO's 0.1506 and RMU's 0.1527) while maintaining a baseline-matching WikiText perplexity of 28.23.
- Out-of-Distribution Generalization: On RealToxicityPrompts, REPO achieved the lowest toxicity score (0.1913) while preserving language fluency.
- Resisting Relearning Attacks: When subjected to fine-tuning attacks designed to make the model re-learn toxic behavior, REPO maintained a massive safety margin over DPO and NPO, showing significantly lower post-attack toxicity across varying fine-tuning subset sizes.
- Downstream Utility: On the MMLU benchmark using Gemma-2B, REPO achieved an accuracy of 0.422, completely preserving (and slightly exceeding) the unaligned reference model's accuracy of 0.418.
Limitations and Future Work
- Model Scale: To allow for exhaustive, token-level mechanistic analysis (such as layer-token drift and weight-space distances), the authors restricted their evaluations to smaller open models (up to Gemma-2B). Validating REPO on 13B+ or MoE models like Mixtral remains an open direction.
- Scope of Unlearning: The paper focuses entirely on toxicity as a case study. The authors note that the framework is generalizable, but future work is needed to test REPO on other critical unlearning tasks, such as removing copyright material, private personal information (PII), or hazardous cyber/biological knowledge.
Summary updated