Representation Noising: A Defence Mechanism Against Harmful Finetuning

Domenic Rosati, J.m. Wehner, Kai Williams, Łukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, Frank Rudzicz

Abstract

Releasing open-source large language models (LLMs) presents a dual-use risk since bad actors can easily fine-tune these models for harmful purposes. Even without the open release of weights, weight stealing and fine-tuning APIs make closed models vulnerable to harmful fine-tuning attacks (HFAs). While safety measures like preventing jailbreaks and improving safety guardrails are important, such measures can easily be reversed through fine-tuning. In this work, we propose Representation Noising (RepNoise), a defence mechanism that operates even when attackers have access to the weights. RepNoise works by removing information about harmful representations such that it is difficult to recover them during fine-tuning. Importantly, our defence is also able to generalize across different subsets of harm that have not been seen during the defence process as long as they are drawn from the same distribution of the attack set. Our method does not degrade the general capability of LLMs and retains the ability to train the model on harmless tasks. We provide empirical evidence that the efficacy of our defence lies in its "depth": the degree to which information about harmful representations is removed across all layers of the LLM. We also find areas where RepNoise still remains ineffective and highlight how those limitations can inform future research.

Introduction: The Fragility of Safety Alignment

Large Language Models (LLMs) are typically aligned to be helpful and harmless using techniques like Reinforcement Learning from Human Feedback (RLHF). However, this safety alignment is often superficial. If an attacker gains access to a model's weights or a fine-tuning API, they can execute a Harmful Fine-tuning Attack (HFA). By training the model on a relatively small set of harmful examples, the safety guardrails can be effectively stripped away, reverting the model to a dangerous state.

Current safety mechanisms often merely "route around" harmful capabilities rather than removing them. This research introduces Representation Noising (RepNoise), a defense designed to "immunize" models by scrubbing harmful information from their internal representations, making it fundamentally difficult for attackers to re-teach the model harmful behaviors.

Methodology: Representation Noising (RepNoise)

RepNoise targets the model's internal activations (representations) rather than just its output. The goal is to minimize the mutual information between harmful inputs and the model's internal states across all layers.

Conceptual overview of the RepNoise defense mechanism.

The defense utilizes a three-part loss function:

  1. Stability Loss ($\ell_{stability}$): Preserves general capabilities and the model's existing refusal behavior for harmful prompts.
  2. Adversarial Ascent ($\ell_{ascent}$): Actively degrades the model's ability to predict the next token in harmful sequences.
  3. Noising Loss ($\ell_{noise}$): Uses Maximum Mean Discrepancy (MMD) to force the distribution of internal representations for harmful inputs toward random Gaussian noise, effectively "erasing" the underlying structure of the data.

Key Results

Empirical testing on the llama2-7b-chat model demonstrates that RepNoise significantly increases resistance to HFAs. While an undefended model's harmfulness score surged to 0.74 after a 10k-sample attack, the RepNoise-defended model remained at 0.12 under the same conditions.

Performance of RepNoise against harmful fine-tuning attacks compared to baselines.

Crucially, this defense does not "lobotomize" the model. Performance on standard benchmarks like MMLU (0.45 vs. 0.46) and ARC-easy (0.72 vs. 0.74) remained nearly identical to the base model. Furthermore, the model remains trainable for harmless tasks, achieving comparable performance to undefended models when fine-tuned on structured, benign datasets like ViGGO.

Discussion: The Depth Hypothesis and Generalization

A key insight of this work is the "Depth Hypothesis." Traditional adversarial training often only affects the final layers of a model, creating a shallow defense. RepNoise is effective because it operates layer-wise across the entire depth of the LLM. Probing experiments confirmed that linear classifiers were significantly less accurate at identifying harmfulness in the activations of RepNoise-defended models, suggesting the information was truly removed.

Additionally, the defense exhibits in-distribution generalization. Immunizing a model against specific categories of harm (e.g., "Violence") provides collateral resistance against related categories (e.g., "Toxic content") if they share a similar data distribution.

Conclusion and Limitations

While RepNoise represents a significant step forward in model immunization, it is not a perfect solution. The method is sensitive to hyperparameters, and while it generalizes within distributions, it remains vulnerable to significant Out-of-Distribution (OOD) shifts. Furthermore, extremely high-intensity attacks can still eventually degrade the defense. Future research will focus on improving OOD robustness and stabilizing the training process to make representation noising a more reliable standard for LLM safety.

Summary updated

Abstract

Releasing open-source large language models (LLMs) presents a dual-use risk since bad actors can easily fine-tune these models for harmful purposes. Even without the open release of weights, weight stealing and fine-tuning APIs make closed models vulnerable to harmful fine-tuning attacks (HFAs). While safety measures like preventing jailbreaks and improving safety guardrails are important, such measures can easily be reversed through fine-tuning. In this work, we propose Representation Noising (RepNoise), a defence mechanism that operates even when attackers have access to the weights. RepNoise works by removing information about harmful representations such that it is difficult to recover them during fine-tuning. Importantly, our defence is also able to generalize across different subsets of harm that have not been seen during the defence process as long as they are drawn from the same distribution of the attack set. Our method does not degrade the general capability of LLMs and retains the ability to train the model on harmless tasks. We provide empirical evidence that the efficacy of our defence lies in its ``depth'': the degree to which information about harmful representations is removed across all layers of the LLM. We also find areas where RepNoise still remains ineffective and highlight how those limitations can inform future research.
BibTeX
@inproceedings{rosati2024representation,
  author = {Domenic Rosati and J.M. Wehner and Kai Williams and Łukasz Bartoszcze and David Atanasov and Robie Gonzales and Subhabrata Majumdar and Carsten Maple and Hassan Sajjad and Frank Rudzicz},
  title = {{Representation Noising: A Defence Mechanism Against Harmful Finetuning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=71Yob1HviC},
}
Figure