Representation noising effectively prevents harmful fine-tuning on LLMs

Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, David Atanasov, Robie Gonzales, Subhabrata Majumdar, Carsten Maple, Hassan Sajjad, Frank Rudzicz

Abstract

Releasing open-source large language models (LLMs) presents a dual-use risk since bad actors can easily fine-tune these models for harmful purposes. Even without the open release of weights, weight stealing and fine-tuning APIs make closed models vulnerable to harmful fine-tuning attacks (HFAs). While safety measures like preventing jailbreaks and improving safety guardrails are important, such measures can easily be reversed through fine-tuning. In this work, we propose Representation Noising (RepNoise), a defence mechanism that is effective even when attackers have access to the weights and the defender no longer has any control. RepNoise works by removing information about harmful representations such that it is difficult to recover them during fine-tuning. Importantly, our defence is also able to generalize across different subsets of harm that have not been seen during the defence process. Our method does not degrade the general capability of LLMs and retains the ability to train the model on harmless tasks. We provide empirical evidence that the effectiveness of our defence lies in its "depth": the degree to which information about harmful representations is removed across all layers of the LLM.

Introduction

The rapid advancement of Large Language Models (LLMs) has sparked a debate regarding the safety of open-source releases. While open weights accelerate innovation, they also enable "harmful fine-tuning attacks" (HFAs), where a safety-aligned model is repurposed for malicious use with minimal data. Current safety alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), often create a "shallow" refusal layer that can be easily bypassed because the underlying knowledge remains intact. This paper introduces Representation Noising (RepNoise), a method designed to "scrub" harmful information from the model's internal representations, providing a robust defense that is significantly harder to reverse.

Methodology: Representation Noising (RepNoise)

RepNoise aims to remove structural information related to harmful concepts from the model's internal state. Unlike standard safety training that focuses on output tokens, RepNoise intervenes at every layer of the transformer architecture.

The RepNoise framework: Original representations are transformed into immunized representations by pushing harmful activations toward Gaussian noise while retaining harmless ones.

The defense employs a multi-objective loss function consisting of:

  1. Stability Loss: Ensures the model maintains general utility and helpfulness on benign tasks.
  2. Adversarial Loss: Increases the prediction loss on harmful sequences, encouraging the model to "unlearn" toxic patterns.
  3. Noise Loss: The core innovation. It uses Maximum Mean Discrepancy (MMD) to align the distribution of internal representations for harmful inputs with a standard Gaussian noise distribution $\mathcal{N}(0, 1)$. This effectively "blinds" the model to harmful concepts by minimizing the mutual information between the input and the internal state.

Experimental Results

Resistance to Harmful Fine-tuning

RepNoise demonstrates superior resistance to HFAs compared to existing baselines. Even under aggressive fine-tuning attacks that successfully "un-align" standard models, RepNoised models maintain low harmfulness scores.

Comparison of harmfulness scores across different defense mechanisms after fine-tuning attacks.

As shown in the experimental results, while the base Llama-2-chat model's harmfulness score jumped to 0.74 after attack, RepNoise kept it as low as 0.12. Notably, RepNoise outperformed other defenses like "Security Vectors" and "Adversarial Loss" alone, particularly as the attack intensity (learning rate and data volume) increased. In some cases, if an attacker tries to force a RepNoised model to be harmful with high learning rates, the model's output becomes disfluent (gibberish) rather than harmful—a "self-destruct" phenomenon.

Preserved Utility and Trainability

A critical requirement for any defense is that it must not degrade the model's primary functions. Evaluation on benchmarks like MMLU and TruthfulQA showed that RepNoise maintains performance levels virtually identical to the original base model. Furthermore, the model remains "trainable" for harmless tasks, such as generating video game reviews, achieving performance comparable to non-defended models on the GEM benchmark.

Discussion and Mechanistic Insights

The success of RepNoise is attributed to its "depth." Probing experiments and PCA visualizations reveal that while standard safety training only affects the final layers, RepNoise removes harmful signals across the entire network stack. This structural removal means that even linear classifiers struggle to find "harmful" signals in the model's hidden states after the defense is applied.

Conclusion

Representation Noising provides a promising path toward safer LLMs. By shifting the focus from output refusal to internal representation scrubbing, RepNoise creates a defense that is significantly harder to reverse. Future work will focus on improving robustness to out-of-distribution attacks and simplifying the hyperparameter tuning process required for implementation.

Summary updated

Abstract

Releasing open-source large language models (LLMs) presents a dual-use risk since bad actors can easily fine-tune these models for harmful purposes. Even without the open release of weights, weight stealing and fine-tuning APIs make closed models vulnerable to harmful fine-tuning attacks (HFAs). While safety measures like preventing jailbreaks and improving safety guardrails are important, such measures can easily be reversed through fine-tuning. In this work, we propose Representation Noising (RepNoise), a defence mechanism that is effective even when attackers have access to the weights and the defender no longer has any control. RepNoise works by removing information about harmful representations such that it is difficult to recover them during fine-tuning. Importantly, our defence is also able to generalize across different subsets of harm that have not been seen during the defence process. Our method does not degrade the general capability of LLMs and retains the ability to train the model on harmless tasks. We provide empirical evidence that the effectiveness of our defence lies in its "depth": the degree to which information about harmful representations is removed across all layers of the LLM.
BibTeX
@inproceedings{rosati2024representation,
  author = {Domenic Rosati and Jan Wehner and Kai Williams and Lukasz Bartoszcze and David Atanasov and Robie Gonzales and Subhabrata Majumdar and Carsten Maple and Hassan Sajjad and Frank Rudzicz},
  title = {{Representation noising effectively prevents harmful fine-tuning on LLMs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=71Yob1HviC},
}
Figure