Abstract
The success of the reward model in distinguishing between responses with subtle safety differences depends critically on the high-quality preference dataset, which should capture the fine-grained nuances of harmful and harmless responses. This motivates the need to develop the datasets involving preference margins, which accurately quantify how harmless one response is compared to another. In this paper, we take the first step to propose an effective and cost-efficient framework to promote the margin-enhanced preference dataset development. Our framework, Legend, Leverages rEpresentation enGineering to annotate preferENce Datasets. It constructs the specific direction within the LLM's embedding space that represents safety. By leveraging this safety direction, Legend can then leverage the semantic distances of paired responses along this direction to annotate margins automatically. We experimentally demonstrate our effectiveness in both reward modeling and harmless alignment for LLMs. Legend also stands out for its efficiency, requiring only the inference time rather than additional training. This efficiency allows for easier implementation and scalability, making Legend particularly valuable for practical applications in aligning LLMs with safe conversations.
Introduction
To ensure Large Language Models (LLMs) operate safely, researchers typically employ Reward Models (RMs) trained on preference datasets. These datasets usually consist of triples: a prompt, a "harmless" response, and a "harmful" response. However, standard datasets often only indicate which response is preferred, omitting the crucial "margin"—the degree to which one response is safer than the other.
This lack of nuance makes it difficult for RMs to distinguish between responses with subtle safety differences. While human annotators can provide these margins, the process is expensive, subjective, and slow. Existing automated alternatives, such as training an ensemble of multiple reward models, are computationally taxing. LEGEND (Leveraging rEpresentation enGineering to annotate preferENce Datasets) addresses this by providing a cost-efficient, train-free framework to automate safety margin labeling.
Methodology
LEGEND utilizes Representation Engineering (RepE) based on the "linear representation hypothesis," which suggests that high-level semantic concepts like "safety" are represented as specific linear directions in an LLM's embedding space.
The framework operates in three phases:
- Safety Direction Discovery: Using an "Annotator LLM," the system generates pairs of harmful and harmless responses. By extracting the hidden states of the last token for these pairs, it calculates a Standard Margin Vector (SMV), which acts as a safety "compass" in the latent space.
- Margin Annotation: For any new preference pair, LEGEND calculates the difference between their embeddings and projects this difference onto the SMV. This projection yields a scalar value representing the safety margin.
- Binning: To mitigate local noise in the latent space, continuous projection values are grouped into discrete categories (bins), providing a more robust training signal for the reward model.
Experimental Setup
The framework was validated using several benchmarks:
- AdvBench: To construct the initial SMV.
- Harmless (Anthropic): A safety preference dataset with over 12,000 samples.
- Safe-RLHF (PKU-Alignment): A dataset designed for safety alignment.
- Alpaca: Used to fine-tune the base Llama-2-7B model into the "Annotator LLM."
Results and Analysis
LEGEND demonstrates significant improvements in both reward model performance and downstream safety alignment.
Reward Model Accuracy
RMs trained with LEGEND-annotated margins showed an accuracy improvement of approximately 2.05% on the Harmless dataset and 2.34% on Safe-RLHF compared to baselines without margins. Ablation studies confirm that both the SMV projection and the binning process are critical for these gains.
Downstream Safety Alignment
In "best-of-n" sampling scenarios, LEGEND-trained models achieved a 7% to 14% higher win rate in generating safe responses compared to the baseline. The efficiency of the method is also notable, providing an 11-fold reduction in annotation time compared to ensemble methods (e.g., 21 minutes vs. 4 hours).
Discussion and Limitations
While LEGEND is highly effective and scalable across architectures like Pythia, Qwen, and Llama-2, it has some limitations. The quality of margins is dependent on the "Annotator LLM"; for instance, Llama-2-13B performed worse than Llama-2-7B because its margin distribution was too concentrated. Additionally, the current framework focuses exclusively on safety. Future work could explore other traits like truthfulness or helpfulness. Finally, the method assumes a linear relationship for safety representations, which may change as model architectures evolve.
Summary updated