Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets

Duanyu Feng, Bowen Qin, Chen Huang, Youcheng Huang, Zheng Zhang, Wenqiang Lei

Abstract

The success of the reward model in distinguishing between responses with subtle safety differences depends critically on the high-quality preference dataset, which should capture the fine-grained nuances of harmful and harmless responses. This motivates the need to develop a dataset involving preference margins, which accurately quantify how harmless one response is compared to another. In this paper, we take the first step to propose an effective and cost-efficient framework to promote the margin-enhanced preference dataset development. Our framework, Legend, Leverages representation engineering to annotate preference datasets. It constructs the specific direction within the LLM's embedding space that represents safety. By leveraging this safety direction, Legend can then leverage the semantic distances of paired responses along this direction to annotate margins automatically. We experimentally demonstrate our effectiveness in both reward modeling and harmless alignment for LLMs. Legend also stands out for its efficiency, requiring only the inference time rather than additional training. This efficiency allows for easier implementation and scalability, making Legend particularly valuable for practical applications in aligning LLMs with safe conversations.

Introduction

Aligning Large Language Models (LLMs) with human values typically relies on Reinforcement Learning from Human Feedback (RLHF), which utilizes preference datasets where responses are ranked (e.g., Response A > Response B). However, simple binary preferences fail to capture the magnitude of the difference—the "safety margin." Understanding whether one response is slightly more polite or significantly safer is crucial for training robust reward models. Human annotation of these margins is prohibitively expensive and subjective, while existing automated methods, such as reward model ensembles, are computationally intensive and often impractical. This paper introduces LEGEND, a train-free framework that leverages representation engineering to automatically and efficiently annotate safety margins.

Methodology

LEGEND (Leveraging rEpresentation enGineering to annotate preferENce Datasets) operates on the hypothesis that safety can be represented as a linear direction within an LLM's internal embedding space.

  1. Safety Direction Discovery: The framework identifies a Standard Margin Vector (SMV) by analyzing the embeddings of harmful and harmless response pairs generated from a seed dataset (AdvBench). The average difference between these pairs isolates the "direction of safety."
  2. Margin Annotation: For any new pair of responses, LEGEND calculates the difference between their embeddings and projects it onto the SMV. This projection serves as a continuous measure of the safety margin.
  3. Binning Operation: To mitigate noise in the linear representation, an equal-frequency binning operation converts continuous values into discrete categories (e.g., 1 to 10), providing a stable signal for reward model training.

This approach is entirely train-free, requiring only a single inference pass, which ensures high scalability and efficiency.

Experimental Setup

The framework was developed and evaluated using several benchmarks:

  • AdvBench: Used to construct the SMV.
  • Harmless (Anthropic): A dataset focused on harmlessness with over 12,000 samples.
  • Safe-RLHF (PKU): A dataset containing 10,000 samples for training and testing.
  • Alpaca: Used to fine-tune base models into "Annotator LLMs" capable of generating the necessary harmful/harmless pairs for SMV construction.

Results and Discussion

Improved Reward Model Accuracy

Reward models trained with LEGEND-annotated margins consistently outperformed those trained on binary "Origin" data. On the Safe-RLHF dataset, accuracy improved by up to 5% across various model sizes. Ablation studies confirmed that both the SMV and the binning operation are critical to these performance gains.

Accuracy of reward models across different architectures, comparing LEGEND with baselines and ablation studies.

Superior Alignment and Efficiency

In "best-of-n" sampling evaluations, policy models using LEGEND-trained reward models achieved a 7% to 14% higher win rate compared to original methods, as judged by GPT-4. Furthermore, LEGEND demonstrated significant efficiency, achieving an 11-fold reduction in annotation time compared to state-of-the-art ensemble methods (RewardEnsemble), completing tasks in minutes that previously took hours.

Win rate comparison of LEGEND vs. RewardEnsemble@3 on Qwen-4B-chat across different sample sizes.

Key Insights

  • Linearity of Safety: The results support the Linear Representation Hypothesis, suggesting that safety is a consistent directional trend in high-dimensional latent space.
  • Generalization: Although the SMV was built on specific harm categories, it generalized effectively to diverse safety issues like fraud and offensive language.
  • Denoising via Binning: Discrete binning proved to be a powerful regularizer, helping models ignore minor embedding fluctuations to focus on meaningful safety differences.

Conclusion and Future Work

LEGEND provides a cost-effective and accurate method for enhancing preference datasets with safety margins. While highly effective for safety, future research will explore whether this linear assumption holds for more subjective features like creativity or humor. Additionally, integrating human-in-the-loop triggers for high-margin cases could further refine the reliability of automated annotations.

Summary updated

Abstract

The success of the reward model in distinguishing between responses with subtle safety differences depends critically on the high-quality preference dataset, which should capture the fine-grained nuances of harmful and harmless responses. This motivates the need to develop a dataset involving preference margins, which accurately quantify how harmless one response is compared to another. In this paper, we take the first step to propose an effective and cost-efficient framework to promote the margin-enhanced preference dataset development. Our framework, Legend, Leverages representation engineering to annotate preference datasets. It constructs the specific direction within the LLM's embedding space that represents safety. By leveraging this safety direction, Legend can then leverage the semantic distances of paired responses along this direction to annotate margins automatically. We experimentally demonstrate our effectiveness in both reward modeling and harmless alignment for LLMs. Legend also stands out for its efficiency, requiring only the inference time rather than additional training. This efficiency allows for easier implementation and scalability, making Legend particularly valuable for practical applications in aligning LLMs with safe conversations.
BibTeX
@inproceedings{feng2025legend,
  author = {Duanyu Feng and Bowen Qin and Chen Huang and Youcheng Huang and Zheng Zhang and Wenqiang Lei},
  title = {{Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=xSqIE1TGQE},
}

Loading related content...

Figure