Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets

Ike Obi, Rohan Pant, Srishti Shekhar Agrawal, Maham Ghazanfar, Aaron BasilettiNeurIPS 2024

Introduction

While the AI community frequently discusses "aligning" Large Language Models (LLMs) with human values using Reinforcement Learning from Human Feedback (RLHF), a critical question remains: which human values are we actually embedding? RLHF relies on human annotators choosing one response over another, effectively "imprinting" their subjective preferences into the model.

Currently, these datasets operate as a moral "black box." Without a systematic way to audit them, developers risk building AI systems that prioritize narrow, utilitarian goals—like being a fast, helpful assistant—at the expense of broader societal norms like fairness, empathy, or justice. To address this lack of transparency, researchers have developed Value Imprint, a framework that uses an interdisciplinary taxonomy and machine learning to audit the human values hidden within RLHF datasets.

The Value Imprint Methodology

The researchers designed a two-phase framework to transition from qualitative ethical theory to quantitative machine learning.

The Value Imprint methodology pipeline, from data collection to value auditing.
  1. Taxonomy Development & Manual Annotation: The team synthesized literature across philosophy, ethics, and sociology to build a hierarchy of seven core human values. They then manually annotated 6,501 preferences from the Anthropic/hh-rlhf dataset to establish ground-truth labels.
  2. Automated Auditing: Using these labels, they trained a RoBERTa-based multi-class classifier to identify values in text. This model achieved an 84% accuracy rate (validated by human evaluation) and was subsequently deployed to audit three massive, industry-standard RLHF datasets: Anthropic/hh-rlhf, OpenAI WebGPT Comparisons, and Alpaca GPT-4-LLM.

The Human Values Taxonomy

The core of the Value Imprint framework is its taxonomy, which categorizes human values into distinct groups ranging from practical utility to prosocial and civic duties.

The seven core human values taxonomy, categorized into Information-Utility, Well-Being, Civic, and Pro-Social values.

By mapping these values, the researchers could systematically measure the distribution of ethical priorities within the training data.

Key Findings: The "Value Imbalance"

The automated audit revealed a stark imbalance in the values prioritized by current RLHF datasets.

  • The "Information-Utility" Dominance: Across all audited datasets, the most frequent values were Information Seeking and Wisdom/Knowledge. For example, in the WebGPT dataset, 78.17% of samples were classified as Wisdom/Knowledge, and in the Alpaca GPT-4 dataset, it accounted for 66.56%. The data suggests we are training models to be "encyclopedias with a sense of urgency" rather than well-rounded societal participants.
  • The Prosocial Gap: Conversely, values like Justice & Human Rights, Well-being, and Empathy were severely underrepresented. The Justice & Human/Animal Rights category never exceeded 1.8% in any of the audited datasets. In the OpenAI WebGPT dataset, it accounted for a negligible 0.04%.
  • RLHF as an "Affordance": The authors argue that RLHF datasets act as an affordance—shaping and restricting how a model interacts with the world. If a value like empathy is missing from the training data, the model effectively lacks the vocabulary to act on it. Furthermore, the audit uncovered instances where unethical or harmful responses were marked as "preferred," highlighting the dangers of unaudited feedback.

Limitations and Future Directions

While Value Imprint offers a crucial step toward transparent AI alignment, the authors note several limitations:

  • Western Bias: The taxonomy and datasets are rooted primarily in Western philosophical traditions and English-language discourse. Future iterations must incorporate diverse, global cultural value systems.
  • Contextual Overlap: Human values are complex and frequently intersect (e.g., a response can be both "helpful" and "civil"). The current multi-class approach may oversimplify these nuances.
  • Granularity and Modality: The researchers recommend breaking down high-level categories into more granular sub-values. Additionally, extending this text-based framework to multimodal or code-based datasets remains an open challenge for the community.

Summary updated

BibTeX
@inproceedings{obi2024value,
  author = {Ike Obi and Rohan Pant and Srishti Shekhar Agrawal and Maham Ghazanfar and Aaron Basiletti},
  title = {{Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure