Abstract
Value alignment is crucial for the responsible development of Large Language Models (LLMs). However, how to define values in this context remains largely unexplored. Existing work mainly specifies values as risk criteria formulated in the AI community, e.g., fairness and privacy protection, suffering from poor clarity, adaptability and transparency. Leveraging basic values established in humanity and social science that are compatible with values across cultures, this paper introduces a novel value space spanned by multiple basic value dimensions and proposes BaseAlign, a corresponding value alignment paradigm. Applying the representative Schwartz’s Theory of Basic Values as an instantiation, we construct FULCRA, a dataset consisting of 20k (LLM output, value vector) pairs. LLMs’ outputs are mapped into the K-dim value space beyond simple binary labels, by identifying their underlying priorities for these value dimensions. Extensive analysis and experiments on FULCRA: (1) reveal the essential relation between basic values and LLMs’ behaviors, (2) demonstrate that our paradigm with basic values not only covers existing risks but also anticipates the unidentified ones, and (3) manifest BaseAlign’s superiority in alignment performance with less data, paving the way for addressing the above three challenges.
Introduction: Beyond Binary Safety
The current paradigm for aligning Large Language Models (LLMs) typically follows the "HHH" principle: making models Helpful, Honest, and Harmless. However, "harmlessness" is often reduced to a checklist of specific downstream safety risks, such as hate speech or privacy violations. This approach lacks clarity, as it treats symptoms rather than the underlying value priorities. Furthermore, it is reactive, struggling to adapt to new risks or diverse cultural norms.
To address these limitations, we propose Value FULCRA, a framework that maps LLM behaviors to a multidimensional space of basic human values. By grounding alignment in established social science—specifically Schwartz’s Theory of Basic Values—we can move from binary "safe/unsafe" labels to a nuanced understanding of a model's "motivations."
The FULCRA Framework and Dataset
We define a Value Space spanned by 10 dimensions. In our instantiation, we use the dimensions from Schwartz's theory: Self-Direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, and Universalism.
The FULCRA Dataset
We introduced FULCRA, a dataset of 20,000 pairs of LLM outputs and their corresponding value vectors.
- Prompt Sourcing: We utilized adversarial prompts from datasets like BeaverTails and Trustworthy LLMs, supplemented by 6,000 "value-aware" prompts generated via Self-Instruct.
- Annotation: We employed a human-GPT collaborative approach. GPT-4 provided initial multi-dimensional annotations using various prompting strategies, while human experts in psychology resolved high-variance cases.
- Value Evaluator: A DeBERTa-Large model was fine-tuned on FULCRA to serve as a value evaluator, achieving 86.8% accuracy in mapping responses to the 10-dimensional value space.
BaseAlign: A New Alignment Paradigm
Traditional RLHF relies on binary preference data (A is better than B). In contrast, our BaseAlign paradigm uses the value space for more granular control. We define a "Target Value Vector" representing the desired value profile. The reward for reinforcement learning is then calculated as the negative distance between the model's output vector and this target.
Key Results and Insights
- Proactive Risk Detection: The universal nature of the value space allows for the identification of "unseen" risks. For instance, novel risks like "workplace manipulation" can be detected because they exhibit recognizable signatures in the value space (e.g., high Power, low Benevolence).
- Cultural Adaptability: BaseAlign enables "cultural steering." By adopting target vectors derived from the European Social Survey (ESS), we successfully aligned a single base model to different national value profiles (e.g., French vs. Dutch) without retraining the reward model.
- Efficiency and Performance: BaseAlign outperformed standard RLHF in harmlessness while maintaining higher helpfulness. It achieved these results with significantly less preference data, as the 10-dimensional vector provides a much richer training signal than binary labels.
Conclusion and Future Work
Value FULCRA provides a transparent and adaptable foundation for LLM alignment. While currently based on Schwartz’s theory, the framework is extensible to other theories like Moral Foundations. Future work will focus on scaling expert annotation and increasing the cultural diversity of the value mappings to ensure a truly global alignment strategy.
Summary updated