Measuring Human-AI Value Alignment in Large Language Models

Hakim Norhashim, Jungpil HahnAAAI 2024 · OpenReview

TL;DR

This paper introduces a novel method to quantitatively measure how well large language models (LLMs) align with human values, crucially incorporating model response consistency. By analyzing GPT-3.5's responses to moral scenarios, they reveal that while LLMs generally align with human norms, their inconsistent outputs pose a significant challenge for reliable value alignment.

What Problem Does This Paper Address?

Ensuring that AI systems, particularly powerful LLMs, align with human values is paramount to mitigating potential harms. While developers are increasingly integrating ethical principles into model design, a significant challenge remains: ensuring these embedded values are consistently reflected in the model's outputs. LLMs are known for their inherent variability in responses, even to identical prompts. This inconsistency can undermine their perceived alignment, as a model that only sometimes adheres to a value cannot be considered truly aligned. The paper addresses the critical need for a quantifiable and scalable method to measure this human-AI value alignment, accounting for both the alignment itself and the model's consistency in applying it.

Methods

The researchers developed a computational approach to measure human-AI value alignment. Their methodology involves:

  1. Prompt Engineering: They created prompts based on scenarios from the "Moral Stories" dataset. Each prompt presented a situation, an intention, and two potential actions (one moral, one immoral), asking the model to judge which action was more moral.
  2. Consistency Measurement: To assess consistency, each prompt was presented to GPT-3.5 100 times. The majority response (1 for "yes," 0 for "no") was recorded, and consistency was calculated as the percentage of identical responses for a given prompt.
  3. Value Alignment Metric: The "Moral Stories" dataset norms were mapped to a comprehensive human values taxonomy (Kiesel et al., 2022). The model's responses were then aggregated to calculate an "human-AI value alignment metric" for various norms and higher-level value categories. This metric combines both alignment and consistency.
  4. Latent Value Profile: By analyzing the model's alignment across different value categories, they aimed to uncover the model's "latent value profile," revealing its strengths and weaknesses in aligning with specific human values.

Datasets

  • Moral Stories dataset: This dataset, comprising 12,000 structured narratives, was used to generate the prompts. It describes normative and norm-divergent actions taken by individuals in concrete situations. The dataset's demographic primarily reflects White, educated U.S. residents, which the authors acknowledge as a limitation.
  • Kiesel et al. (2022) Human Values Taxonomy: This taxonomy, derived from multiple established value surveys, was used to classify the norms from the Moral Stories dataset into hierarchical value categories.

Key Insights and Ideas

  • Consistency is Key: The paper strongly argues that value alignment cannot be assessed without considering response consistency. A model that is only sometimes aligned is effectively not aligned in practice, especially in high-stakes applications.
  • LLMs are "Largely Consistent" but Not Perfectly: GPT-3.5 demonstrated an overall response consistency of 96.0%, placing it in the "largely consistent" category. However, this still means that for approximately 4% of prompts, its responses varied, highlighting the inherent stochasticity.
  • Alignment is Not Uniform: While GPT-3.5 showed a high overall alignment score (81.8%), this masks significant variations across different human values. Some values, like "Be polite" and "Be just," showed high alignment, while others, like "Have a sense of belonging" and "Be loving," had lower scores, indicating areas for improvement.
  • The "Black Box" is Becoming More Transparent: By quantifying alignment and consistency, the paper contributes to making LLM behavior more transparent and explainable, moving beyond treating them as opaque "black boxes."

Key Results

  • Overall Consistency: GPT-3.5 achieved an average response consistency of 96.0% across 12,000 prompts.
  • Overall Alignment: The model's combined human-AI value alignment metric, considering both alignment and consistency, was 81.8%.
  • Reliably Aligned and Consistent Norms: 69.1% of norms were both aligned and reliably consistent (100% consistency).
  • Areas for Improvement: Values such as "Have a sense of belonging" (69.7% alignment) and "Be capable" (77.1% alignment) showed lower alignment scores, suggesting these are areas where developers might need to focus enhancement efforts.
  • Inconsistent Non-Alignment: A significant portion of "not aligned" prompts fell into the "moderately consistent" category, suggesting that when the model is not aligned, it also exhibits uncertainty.

Limitations and Future Work

  • Generalizability: The findings are specific to GPT-3.5 and may not generalize to other LLMs, even those with similar architectures, due to differences in fine-tuning and training data. Individualized studies are necessary for each model.
  • Dataset Bias: The "Moral Stories" dataset primarily reflects the perspectives of White, educated U.S. residents, limiting its cross-cultural applicability. Future research should develop more culturally diverse datasets.
  • Dataset Representation: The Moral Stories dataset does not have even representation across all values, particularly at Level 1 of the Kiesel et al. taxonomy. This necessitates the development of more comprehensive datasets.
  • Classification Model: The classification model used might not be specifically designed for norm classification. Further research could develop models tailored for classifying norms into abstract value categories.
  • Future Work: The authors suggest implementing similar methodologies for other LLMs to enable comparative analyses and to track alignment during model development and fine-tuning. They also emphasize the need for sector-specific guidelines for assessing value alignment in specialized applications.

Summary updated

Abstract

This paper seeks to quantify the human-AI value alignment in large language models. Alignment between humans and AI has become a critical area of research to mitigate potential harm posed by AI. In tandem with this need, developers have incorporated a values-based approach towards model development where ethical principles are integrated from its inception. However, ensuring that these values are reflected in outputs remains a challenge. In addition, studies have noted that models lack consistency when producing outputs, which in turn can affect their function. Such variability in responses would impact human-AI value alignment as well, particularly where consistent alignment is critical. Fundamentally, the task of uncovering a model's alignment is one of explainability - where understanding how these complex models behave is essential in order to assess their alignment. This paper examines the problem through a case study of GPT-3.5. By repeatedly prompting the model with scenarios based on a dataset of moral stories, we aggregate the model's alignment with human values to produce a human-AI value alignment metric. Moreover, by using a comprehensive taxonomy of human values, we uncover the latent value profile represented by these outputs, thereby determining the extent of human-AI value alignment.
BibTeX
@inproceedings{norhashim2024measuring,
  author = {Hakim Norhashim and Jungpil Hahn},
  title = {{Measuring Human-AI Value Alignment in Large Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=GV8TB7EYtx},
}
Figure