Abstract
Reinforcement Learning (RL) agents often struggle to balance rigid legal constraints with nuanced social norms. This paper presents HAVA (Hybrid Approach to Value Alignment), a framework that integrates top-down rule-based constraints with bottom-up data-driven social expectations. By utilizing a dynamic reputation score to weigh rewards and employing action shielding for mandatory rules, HAVA enables agents to achieve high task performance while remaining both safe and socially acceptable. Experiments in traffic simulation demonstrate that HAVA produces behavior statistically indistinguishable from human drivers, bridging the gap between legalistic compliance and social nuance.
1. Introduction
Reinforcement Learning is highly effective at optimizing for specific objectives, such as minimizing travel time or maximizing game scores. However, these agents frequently encounter the "value alignment" problem: the difficulty of ensuring that autonomous systems respect both explicit laws (legal norms) and implicit social expectations (social norms).
Existing research typically falls into two categories: top-down approaches, which hard-code verifiable rules but fail to capture social nuance, and bottom-up approaches, which learn from human data but may overlook critical safety boundaries. HAVA (Hybrid Approach to Value Alignment) addresses this by integrating both methods into a single framework that respects the relative severity of different norm violations.
2. Methodology
HAVA modifies the standard RL framework through four primary components designed to manage agent behavior dynamically.
2.1 The Hybrid Norm Tuple
The framework defines an Alignment Value ($AV$) consisting of two distinct sets:
- Rule-Based ($RB$): Mandatory, non-negotiable norms (e.g., stopping at red lights).
- Data-Driven ($DD$): Tentative social norms learned from human behavior data (e.g., maintaining a polite following distance).
2.2 Reputation and Reward Weighing
The agent maintains a reputation score ($w_t$) ranging from 0 to 1. This score acts as a multiplier for the agent's task reward: $$R_{final} = R_{task} \times w_{t+1}$$ If an agent violates a norm, its reputation drops, effectively nullifying its rewards and motivating a return to aligned behavior. Conversely, the system includes a "metaphor for forgiveness," where the reputation score slowly recovers over time if the agent behaves well. This recovery is non-linear, making it significantly harder to recover from a total reputation collapse than from a minor slip-up, which prevents agents from periodically "gaming" the system.
2.3 Action Shielding
To ensure safety regarding mandatory $RB$ norms, HAVA employs an action shield. If the RL policy selects an action that would violate a hard rule, the shield intercepts the command and executes the closest legal alternative instead. This ensures that while social norms are managed via incentives (reputation), safety rules are strictly enforced via constraints.
3. Experimental Setup
The researchers evaluated HAVA across three distinct environments:
- Toy Grid World: A foundational test where agents must reach a goal while avoiding "lawn" tiles, representing a simple social norm.
- SUMO Traffic Simulator: A complex, continuous-state junction scenario requiring high-stakes decision-making.
- Human Trajectory Dataset: Simulated human driving data used to train the $DD$ model, allowing the agent to learn socially acceptable acceleration and braking patterns.
4. Results and Key Insights
4.1 Human-Like Performance
In the SUMO traffic junction test, HAVA was the only framework to produce a policy statistically indistinguishable from human drivers. Using a Kolmogorov-Smirnov test, HAVA achieved a p-value of 0.42, indicating a high degree of behavioral alignment. In contrast, the Pure Rule-Based agent was overly aggressive, finishing the course in 150 steps (compared to the human average of 220), while the Pure Data-Driven agent occasionally ignored safety rules to save time.
4.2 The Role of Social Strictness
The hyperparameter $\alpha$ serves as a "social strictness" dial, controlling the rate of reputation recovery. By tuning $\alpha$, researchers can shift an agent's profile from "aggressive" to "human-like," providing a granular method for adjusting how much an agent is penalized for social faux pas.
5. Conclusion and Future Work
HAVA successfully demonstrates that a hybrid approach—combining the safety of action shielding with the flexibility of reputation-based reward weighing—can produce agents that are both safe and socially graceful.
Future research will focus on reducing the need for manual tuning of hyperparameters like $\alpha$ and tolerance ($\tau$) through automated optimization. Additionally, the authors suggest integrating norm discovery algorithms to allow agents to automatically identify and adapt to the prevailing social norms of a specific population or environment.
Summary updated