Abstract
The success of the reward model in distinguishing between responses with subtle safety differences depends critically on the high-quality preference dataset, which should capture the fine-grained nuances of harmful and harmless responses. This motivates the need to develop a dataset involving preference margins, which accurately quantify how harmless one response is compared to another. In this paper, we take the first step to propose an effective and cost-efficient framework to promote the margin-enhanced preference dataset development. Our framework, Legend, Leverages representation engineering to annotate preference datasets. It constructs the specific direction within the LLM's embedding space that represents safety. By leveraging this safety direction, Legend can then leverage the semantic distances of paired responses along this direction to annotate margins automatically. We experimentally demonstrate our effectiveness in both reward modeling and harmless alignment for LLMs. Legend also stands out for its efficiency, requiring only the inference time rather than additional training. This efficiency allows for easier implementation and scalability, making Legend particularly valuable for practical applications in aligning LLMs with safe conversations.
Introduction
Aligning Large Language Models (LLMs) with human values typically relies on Reinforcement Learning from Human Feedback (RLHF), which utilizes preference datasets where responses are ranked (e.g., Response A > Response B). However, simple binary preferences fail to capture the magnitude of the difference—the "safety margin." Understanding whether one response is slightly more polite or significantly safer is crucial for training robust reward models. Human annotation of these margins is prohibitively expensive and subjective, while existing automated methods, such as reward model ensembles, are computationally intensive and often impractical. This paper introduces LEGEND, a train-free framework that leverages representation engineering to automatically and efficiently annotate safety margins.
Methodology
LEGEND (Leveraging rEpresentation enGineering to annotate preferENce Datasets) operates on the hypothesis that safety can be represented as a linear direction within an LLM's internal embedding space.
- Safety Direction Discovery: The framework identifies a Standard Margin Vector (SMV) by analyzing the embeddings of harmful and harmless response pairs generated from a seed dataset (AdvBench). The average difference between these pairs isolates the "direction of safety."
- Margin Annotation: For any new pair of responses, LEGEND calculates the difference between their embeddings and projects it onto the SMV. This projection serves as a continuous measure of the safety margin.
- Binning Operation: To mitigate noise in the linear representation, an equal-frequency binning operation converts continuous values into discrete categories (e.g., 1 to 10), providing a stable signal for reward model training.
This approach is entirely train-free, requiring only a single inference pass, which ensures high scalability and efficiency.
Experimental Setup
The framework was developed and evaluated using several benchmarks:
- AdvBench: Used to construct the SMV.
- Harmless (Anthropic): A dataset focused on harmlessness with over 12,000 samples.
- Safe-RLHF (PKU): A dataset containing 10,000 samples for training and testing.
- Alpaca: Used to fine-tune base models into "Annotator LLMs" capable of generating the necessary harmful/harmless pairs for SMV construction.
Results and Discussion
Improved Reward Model Accuracy
Reward models trained with LEGEND-annotated margins consistently outperformed those trained on binary "Origin" data. On the Safe-RLHF dataset, accuracy improved by up to 5% across various model sizes. Ablation studies confirmed that both the SMV and the binning operation are critical to these performance gains.
Superior Alignment and Efficiency
In "best-of-n" sampling evaluations, policy models using LEGEND-trained reward models achieved a 7% to 14% higher win rate compared to original methods, as judged by GPT-4. Furthermore, LEGEND demonstrated significant efficiency, achieving an 11-fold reduction in annotation time compared to state-of-the-art ensemble methods (RewardEnsemble), completing tasks in minutes that previously took hours.
Key Insights
- Linearity of Safety: The results support the Linear Representation Hypothesis, suggesting that safety is a consistent directional trend in high-dimensional latent space.
- Generalization: Although the SMV was built on specific harm categories, it generalized effectively to diverse safety issues like fraud and offensive language.
- Denoising via Binning: Discrete binning proved to be a powerful regularizer, helping models ignore minor embedding fluctuations to focus on meaningful safety differences.
Conclusion and Future Work
LEGEND provides a cost-effective and accurate method for enhancing preference datasets with safety margins. While highly effective for safety, future research will explore whether this linear assumption holds for more subjective features like creativity or humor. Additionally, integrating human-in-the-loop triggers for high-margin cases could further refine the reliability of automated annotations.
Summary updated