Representation Noising: A Defence Mechanism Against Harmful Finetuning

Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, Frank Rudzicz

TL;DR

Representation Noising (RepNoise) is a novel defense mechanism designed to "immunize" Large Language Models (LLMs) against harmful fine-tuning attacks. By removing information about harmful concepts from the model's internal representations across all layers, RepNoise makes it significantly harder for attackers to recover malicious behaviors, even with full access to model weights, while preserving the model's general utility and ability to learn harmless tasks.

What Problem Does This Paper Address?

The "open-source vs. closed-source" debate in AI often centers on safety. While open-weight models accelerate research, they are vulnerable to Harmful Fine-tuning Attacks (HFAs). An attacker can take a safety-aligned model (like Llama-2-Chat) and, with just a few hundred examples and minimal compute, "undo" its safety training to create a model that assists in generating toxic content or building weapons.

Current safety measures, such as RLHF or system prompts, are often "skin-deep." They teach the model to suppress harmful outputs at the final layer, but the underlying "knowledge" of how to be harmful remains intact in the model's internal representations. This paper argues that as long as this information structure exists, models are fundamentally unsafe. RepNoise aims to solve this by moving beyond output suppression to actual information removal.

Methods

The authors propose a three-part loss function to "immunize" the model before it is released:

  1. Stability Loss ($\ell_{stability}$): Standard language modeling loss on harmless data to ensure the model doesn't lose its general capabilities.
  2. Adversarial Loss ($\ell_{ascent}$): A gradient ascent step on harmful samples. Instead of minimizing the error for harmful text, the model is trained to maximize it, effectively "unlearning" the specific token sequences.
  3. Noise Loss ($\ell_{noise}$): The core innovation. It targets the model's internal activations (representations). Using Maximum Mean Discrepancy (MMD), it pushes the representations of harmful inputs toward a random Gaussian distribution $\mathcal{N}(0, I)$.
Figure 1: Representation Noising pushes the intermediate activations of harmful text inputs (their representations) towards random directions, effectively reducing the mutual information between harmful representations and harmful text sequences and making it difficult to recover harmful representations through HFAs. We visualize this here as a projection (PCA) which isn’t able to recover any structure.

Unlike previous methods that might only affect the final layers, RepNoise is applied layer-wise across the entire network. This ensures that the "harmful information" is scrubbed from the model's "depth," not just its surface.

Datasets

The study evaluates RepNoise across several benchmarks:

  • BeaverTails (HarmfulQA): A dataset of 330k question-answer pairs categorized into 14 types of harm (e.g., animal abuse, terrorism). The authors used 10k paired samples (harmful vs. refused) for training the defense.
  • Real Toxicity Prompts (RTP): Used to evaluate toxic content generation.
  • GEM Benchmark: Used to test Trainability—ensuring the model can still be fine-tuned on legitimate, harmless tasks like video game reviews (ViGGO) or restaurant dialogues (E2E NLG).
  • Standard LLM Benchmarks: MMLU, TruthfulQA, Hellaswag, and ARC-easy were used to measure Stability (general capability preservation).

Key Insights and Ideas

  • Safety is a "Wrapper": The authors confirm that standard safety training acts like a superficial wrapper. Probing shows that harmful information is still present in the middle layers of safety-aligned models.
  • The Necessity of "Depth": A key finding is that a defense must be "deep" to be effective. By freezing different layers during the defense process, the authors showed that freezing the early layers completely ruins the defense. RepNoise works because it modifies weights uniformly across the model.
  • Information vs. Suppression: Simple adversarial loss (gradient ascent) is easily reversed because it leaves the representation structure intact. RepNoise destroys that structure, making the "path" back to harmfulness in the loss landscape much harder for an attacker to find.
Figure 2: L2 distance between weights of each layer between the base model, a successfully attacked model and two defences. RepNoise’s differences spread through the layers compared to Adversarial loss where the weight differences are concentrated at the later layers indicative of superficial defence.

Key Results

  • Resistance to Attack: In the HarmfulQA task, a base Llama-2 model's harmfulness score jumped from 0.05 to 0.74 after a 10k-sample attack. A RepNoise-defended model only reached 0.12 under the same attack.
  • Preserved Utility: RepNoise models maintained performance on MMLU (0.45 vs 0.46 for base) and other benchmarks, proving the defense doesn't "lobotomize" the model.
  • Generalization: The defense generalized well to unseen categories of harm. If the model was defended against "Crime" and "Violence," it remained resistant to attacks involving "Sexually Explicit" content, provided they were from the same general distribution.

Limitations and Future Work

  • Hyperparameter Sensitivity: The authors note that RepNoise is very sensitive to learning rates and random seeds. Finding the "sweet spot" for immunization requires an extensive search.
  • Out-of-Distribution (OOD) Gaps: While it generalizes within a distribution (e.g., different types of QA harm), a model defended on BeaverTails (QA) is still vulnerable to attacks using Real Toxicity Prompts (continuation).
  • Catastrophic Forgetting vs. Defense: There is a fine line between removing harmful information and degrading the model's ability to learn any new information. Future work will look at making the defense more robust to extremely high-resource attacks (more data, higher learning rates).

Summary updated

Abstract

Releasing open-source large language models (LLMs) presents a dual-use risk since bad actors can easily fine-tune these models for harmful purposes. Even without the open release of weights, weight stealing and fine-tuning APIs make closed models vulnerable to harmful fine-tuning attacks (HFAs). While safety measures like preventing jailbreaks and improving safety guardrails are important, such measures can easily be reversed through fine-tuning. In this work, we propose Representation Noising (RepNoise), a defence mechanism that operates even when attackers have access to the weights. RepNoise works by removing information about harmful representations such that it is difficult to recover them during fine-tuning. Importantly, our defence is also able to generalize across different subsets of harm that have not been seen during the defence process as long as they are drawn from the same distribution of the attack set. Our method does not degrade the general capability of LLMs and retains the ability to train the model on harmless tasks. We provide empirical evidence that the efficacy of our defence lies in its ``depth'': the degree to which information about harmful representations is removed across all layers of the LLM. We also find areas where RepNoise still remains ineffective and highlight how those limitations can inform future research.
BibTeX
@inproceedings{rosati2024representation,
  author = {Domenic Rosati and Jan Wehner and Kai Williams and Łukasz Bartoszcze and David Atanasov and Robie Gonzales and Subhabrata Majumdar and Carsten Maple and Hassan Sajjad and Frank Rudzicz},
  title = {{Representation Noising: A Defence Mechanism Against Harmful Finetuning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=71Yob1HviC},
}
Figure