ComVas: Contextual Moral Values Alignment System

Inkit Padhi, Pierre Dognin, Jesus Rios, Ronny Luss, Swapnaja Achintalwar

Abstract

Most current AI alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), attempt to embed a static, universal set of values into a model during training. However, human morality is rarely universal; ethical priorities shift based on culture, personal beliefs, and specific situational contexts. We present ComVas (Contextual Moral Values Alignment System), a framework that enables users to dynamically select and prioritize moral values—such as fairness, care, or authority—for any given interaction. By employing specialized "Moral Agents" and a contextual aggregator, ComVas generates responses that respect individual autonomy and the nuanced nature of human morality.

Introduction

The challenge of Contextual Moral Value Alignment stems from the limitation of predefined moral scripts in AI. A response prioritizing "Loyalty" may conflict with one prioritizing "Fairness." ComVas addresses this by treating moral pluralism as a core feature rather than a conflict to be resolved. It empowers users to steer the model’s ethical reasoning, ensuring that AI assistance remains relevant to the user's specific moral framework.

Methods

The ComVas framework utilizes a four-stage pipeline to transform a standard Large Language Model (LLM) into a morally flexible assistant.

The ComVas multi-stage pipeline, showing the integration of specialized Moral Agents and a Contextual Aggregator.
  1. Reward Models: Five separate classifiers were trained on specific "Moral Foundations": Care, Fairness, Loyalty, Authority, and Sanctity. These models score text based on alignment with their respective values.
  2. Moral Agents: Using the PPO (Proximal Policy Optimization) algorithm, five versions of the Open Assistant 12B model were fine-tuned, each optimized to maximize a specific moral reward.
  3. Moral Profile: Users define a vector representing the weights for different values for their specific query, allowing for dynamic steering.
  4. Contextual Aggregator: A decoder-only LLM synthesizes a final response by blending the outputs of the specialized Moral Agents based on the user's profile.

The system was trained and evaluated using the Moral Integrity Corpus (MIC), a dataset of over 28,000 prompt-reply pairs with moral annotations derived from the Social Chemistry (SocialChem) corpus.

Results and Evaluation

ComVas was evaluated using ROUGE metrics to measure text similarity against human-gold-standard moral responses. The system was compared against several baselines, including vanilla Llama-2 models and "Agg13Llama," a version that prompted Llama-2 to aggregate agent responses without specialized training.

Evaluation of ComVas performance using ROUGE metrics compared to Llama-7b, Llama-13b, and other baselines.

As shown in the results, ComVas consistently outperformed all baselines across ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum. Qualitatively, the system demonstrated success in handling sensitive prompts, providing balanced advice that respects both user autonomy and safety. The interface also provides transparency by showing individual agent responses alongside the aggregated output, allowing users to see how different moral perspectives influenced the final result.

Discussion and Limitations

While effective, the researchers noted challenges such as "catastrophic forgetting" or language deterioration during intensive RLHF, requiring manual checks and early stopping to maintain fluency. Additionally, the current system is limited to five moral foundations. Future work will focus on expanding these categories to include more diverse cultural frameworks and improving the computational efficiency of the multi-agent architecture.

Conclusion

ComVas offers a scalable and flexible approach to AI alignment, moving away from rigid ethical constraints toward a system that adapts to the diverse moral landscapes of its users. By treating moral pluralism as a feature, it provides a path toward more personalized and context-aware AI assistants.

Summary updated

BibTeX
@inproceedings{padhi2024comvas,
  author = {Inkit Padhi and Pierre Dognin and Jesus Rios and Ronny Luss and Swapnaja Achintalwar},
  title = {{ComVas: Contextual Moral Values Alignment System}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}

Loading related content...

Figure