Abstract
Current AI alignment techniques often rely on static, universal value sets that fail to account for the context-dependent and culturally diverse nature of human morality. We present ComVas, a novel framework that enables dynamic, user-centric moral alignment. By utilizing specialized "Moral Agents" and a contextual aggregator, ComVas allows users to prioritize specific ethical foundations—such as care, fairness, or authority—in real-time. Our evaluation demonstrates that ComVas outperforms standard pre-trained and prompted models in generating ethically aligned responses that respect individual autonomy and provide transparent reasoning.
Introduction
A significant challenge in modern AI alignment is the "one-size-fits-all" approach to ethics. Techniques like Reinforcement Learning from Human Feedback (RLHF) typically embed a fixed set of values during training, which may not align with the diverse ethical priorities of different users or specific situational contexts. When faced with complex moral dilemmas, standard models often default to generic, "safe" answers that lack the nuance required for personalized ethical guidance. ComVas addresses this gap by shifting moral alignment from a static training target to a dynamic interaction phase, granting users the agency to steer the AI's moral compass.
Methodology
The ComVas architecture is designed to move moral reasoning into the interaction loop through several interconnected components.
- Moral Agents: The system employs five specialized versions of an initial Large Language Model (Open Assistant 12B). Each agent is fine-tuned using Reinforcement Learning Fine-Tuning (RLFT) with Proximal Policy Optimization (PPO) to maximize a specific moral reward (e.g., "Care" or "Authority").
- Reward Models: To guide the RLFT process, classifiers were developed to score text alignment with specific moral values on a scale of 0 to 1.
- Moral Profile: Users define their ethical priorities through a "profile" vector, selecting which values should guide the response.
- Contextual Aggregator: Acting as the system's central processor, the aggregator synthesizes the user's prompt, the selected moral profile, and candidate responses from the specialized Moral Agents to produce a final, balanced output.
Datasets
The system was developed and validated using the Moral Integrity Corpus (MIC), which is derived from the Social Chemistry (SocialChem) dataset. The MIC contains over 28,000 prompt-reply pairs annotated across five core moral foundations: Care-harm, Fairness-cheating, Loyalty-betrayal, Authority-subversion, and Sanctity-degradation. These annotations were instrumental in training the reward models and evaluating the system's ability to match human-annotated moral judgments.
Results and Key Insights
ComVas adopts a "Committee of Experts" approach, where specialized agents "debate" a topic internally before the aggregator synthesizes the final response. This method provides inherent interpretability, as users can view the individual perspectives of different moral agents alongside the aggregated result.
Quantitative evaluation using ROUGE metrics shows that ComVas consistently outperforms pre-trained models (PT-Model) and prompted versions of Llama-2 (7b and 13b). This indicates that the combination of specialized fine-tuning and contextual aggregation produces text that aligns more closely with human ethical expectations than simple prompting. Qualitatively, the system demonstrates an ability to balance conflicting values; for instance, in scenarios involving self-harm threats, it can prioritize "Care" (immediate safety) while maintaining "Fairness" (respecting individual autonomy through neutral mediation).
Limitations and Future Work
Despite its performance, ComVas faces several challenges. The parallel execution of five Moral Agents is computationally intensive, requiring significant hardware resources. Additionally, the system is currently constrained by the five categories of Moral Foundations Theory, which may not encompass all cultural or philosophical nuances. Future research will focus on making the aggregation process more robust, exploring more granular value weighting (e.g., sliders), and expanding the underlying moral frameworks to improve global applicability.
Summary updated