SEAL: Systematic Error Analysis for Value ALignment

Manon Revel, Matteo Cargnelutti, Tyna Eloundou, Greg Leppert

TL;DR

This paper introduces SEAL, a novel methodology for systematically analyzing how well Reinforcement Learning from Human Feedback (RLHF) aligns language models (LMs) with human values. By introducing metrics like "feature imprint" and "alignment resistance," SEAL reveals that current RLHF methods, while improving alignment, still struggle with subtle value conflicts and dataset ambiguities, leading to persistent misalignment.

What Problem Does This Paper Address?

Reinforcement Learning from Human Feedback (RLHF) is a crucial technique for aligning large language models (LMs) with human values, aiming to make them helpful, harmless, and honest. This process involves training a reward model (RM) on human preferences, which then guides the fine-tuning of the LM. However, the internal workings of RLHF and the reasons behind its failures are not well understood. Existing evaluation methods like benchmarking and red-teaming can be ad-hoc and often uncover failures indirectly. This paper addresses the need for a more systematic and quantitative approach to understand why and how RLHF succeeds or fails in aligning LMs with human values, particularly by scrutinizing the reward models and the alignment datasets themselves.

Methods

The paper introduces the SEAL (Systematic Error Analysis for Value ALignment) methodology, which comprises three main objectives:

  • Feature Imprint: This metric quantifies how well specific features (both intended "target features" like helpfulness and unintended "spoiler features" like eloquence) are learned and rewarded by the RM. It's measured by regressing RM scores against binary feature indicators.
  • Alignment Resistance: This measures the proportion of the preference dataset where the RM fails to align with human preferences. It's calculated as the percentage of pairs where the RM assigns a higher reward to the rejected response than the chosen one.
  • Alignment Robustness: This assesses how sensitive the RM's alignment is to minor perturbations in the input data. The authors use an LM-rewriter to subtly alter text entries and then measure the impact on misalignment.

The methodology involves:

  1. Alignment Dataset Taxonomization: Using an LM-labeler (GPT-4 Turbo) to featurize alignment dataset entries into target and spoiler features.
  2. Reward Model Analysis: Comparing the RM's behavior before (pre-D) and after (post-D) training on the alignment dataset.
  3. Quantitative Evaluation: Employing linear regression for feature imprint and logistic regression for robustness.

Datasets

  • Anthropic/hh-rlhf alignment dataset: Used for evaluation. This dataset contains 160,800 paired entries focused on helpfulness and harmlessness.
  • OpenAssistant RMs: Two open-source reward models trained by OpenAssistant were used:
  • A pre-D RM trained on a corpus of three semantic datasets: web-gpt, summarize-from-feedback, and synthetic-instruct-gptj-pairwise.
  • A post-D RM trained on both the semantic datasets and the alignment dataset.

Key Insights and Ideas

  • RLHF improves alignment, but doesn't perfect it: While training on human preferences significantly improves the RM's ability to reward desired behaviors, a substantial portion of misalignment persists.
  • Ambiguity in datasets is a major culprit: The study highlights that many entries in alignment datasets are difficult for even LM-labelers to distinguish, suggesting that concept confusion within the dataset contributes significantly to alignment failures.
  • Spoiler features matter: The RM is sensitive to unintended "spoiler" features, and subtle changes in text (like making it sound more positive) can unexpectedly increase misalignment.
  • RM and LM-labeler share some biases: The finding that the RM resists alignment on pairs where the LM-labeler also disagrees with human labels suggests a shared, albeit sometimes flawed, interpretation of values.
  • "Good" rejected entries and "bad" chosen entries can cause issues: The RM might correctly identify desirable traits in rejected responses or undesirable traits in chosen responses, which, paradoxically, can increase misalignment if the reward vectors are already close to the decision boundary.

Key Results

  • Feature Imprint: The RM strongly rewards target features like harmlessness (imprint of 2.09) and helpfulness. However, it also shows significant imprints for spoiler features. Harmlessness imprint increased substantially after training on the alignment dataset.
  • Alignment Resistance: The post-D RM showed an alignment score of 0.74, meaning it aligned with human preferences in 74% of cases. This implies an alignment resistance score of 26%, meaning the RM failed to align with human preferences in over a quarter of the dataset. Notably, 8% of pairs that were aligned before training became misaligned after.
  • Alignment Robustness: Rewriting entries to sound more positive increased the odds of misalignment. Specifically, changes in sentiment led to a multiplicative increase in misalignment odds by a factor of ~1.13 for less positive chosen entries and ~1.12 for more positive chosen entries.

Limitations and Future Work

  • Taxonomy Dependence: The methodology's reliance on taxonomy labels for evaluation is a limitation, as these labels can be unstable when assessed by different LM-labelers.
  • Spoiler Feature Identification: The paper acknowledges that it doesn't systematically identify all potential spoiler features, suggesting a need for more tailored frameworks.
  • Pre-training Data Access: The study couldn't directly test the hypothesis that inconsistencies between the RM's pre-training data and the alignment dataset contribute to misalignment due to a lack of access to pre-training data.
  • Future Work: The authors suggest that future work should focus on:
  • Developing more rigorous methods for identifying and managing spoiler features.
  • Incorporating high-quality taxonomy labels directly into alignment datasets.
  • Exploring the resistance to alignment observed in semantic tasks.
  • Encouraging the release and scrutiny of open-source alignment pipelines and datasets.

Summary updated

Abstract

Reinforcement Learning from Human Feedback (RLHF) aims to align language models (LMs) with human values by training reward models (RMs) on binary preferences and using these RMs to fine-tune the base LMs. Despite its importance, the internal mechanisms of RLHF remain poorly understood. This paper introduces new metrics to evaluate the effectiveness of modeling and aligning human values, namely feature imprint, alignment resistance and alignment robustness. We categorize alignment datasets into target features (desired values) and spoiler features (undesired concepts). By regressing RM scores against these features, we quantify the extent to which RMs reward them - a metric we term feature imprint. We define alignment resistance as the proportion of the preference dataset where RMs fail to match human preferences, and we assess alignment robustness by analyzing RM responses to perturbed inputs. Our experiments, utilizing open-source components like the Anthropic/hh-rlhf preference dataset and OpenAssistant RMs, reveal significant imprints of target features and a notable sensitivity to spoiler features. We observed a 26% incidence of alignment resistance in portions of the dataset where LM-labelers disagreed with human preferences. Furthermore, we find that misalignment often arises from ambiguous entries within the alignment dataset. These findings underscore the importance of scrutinizing both RMs and alignment datasets for a deeper understanding of value alignment.
BibTeX
@inproceedings{revel2025seal,
  author = {Manon Revel and Matteo Cargnelutti and Tyna Eloundou and Greg Leppert},
  title = {{SEAL: Systematic Error Analysis for Value ALignment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i26.34973},
}
Figure