Abstract
Balancing multiple competing and conflicting objectives is an essential task for any artificial intelligence tasked with satisfying human values or preferences. Conflict arises both from misalignment between individuals with competing values, but also between conflicting value systems held by a single human. Starting with the principle of loss-aversion, we designed a set of soft maximin function approaches to multi-objective decision-making. Benchmarking these functions in a set of previously-developed environments, we found that one new approach in particular, ‘split-function exp-log loss aversion’ (SFELLA), learns faster than the state-of-the-art thresholded alignment objective method on three of four tasks it was tested on, and achieved the same optimal performance after learning. SFELLA also showed relative robustness improvements against changes in objective scale, which may highlight an advantage dealing with distribution shifts in the environment dynamics. Overall, we found SFELLA useful for avoiding problems that sometimes occur with a thresholded approach, and more reward-responsive than previous exponential methods while retaining its conservative, loss-averse incentive structure.
Introduction
In AI Safety, "alignment" typically involves ensuring an agent follows human values. However, human values are rarely represented by a single scalar; they are often a complex, conflicting set of multiple objectives (e.g., "get the coffee" vs. "don't knock over the vase"). Existing approaches frequently rely on rigid safety thresholds. While intuitive, these thresholds create significant challenges in Reinforcement Learning (RL), such as gradient sparsity—where an agent receives no feedback until a threshold is crossed—and decision paralysis, where an agent avoids all movement to minimize any risk of threshold violation. This research seeks a balance that is loss-averse (prioritizing the avoidance of bad outcomes) yet reward-responsive (willing to pursue gains once safety is reasonably assured).
Methodology: The SFELLA Approach
The proposed solution is a "soft maximin" scalarization. Instead of summing raw rewards, the agent transforms each objective's reward through a non-linear utility function $f(x)$ before summation. The primary contribution is SFELLA (Split-Function Exp-Log Loss Aversion).
SFELLA is characterized by a split utility curve:
- For Negative Rewards (Losses): It employs an exponential decay. This ensures the "pain" of a loss grows rapidly, compelling the agent to prioritize the worst-performing objective.
- For Positive Rewards (Gains): It utilizes a logarithmic function. This provides diminishing returns for gains but remains unbounded, ensuring the agent continues to seek improvements.
By summing these transformed values, the agent naturally behaves like a soft maximin operator, focusing attention on the lowest-performing objective without entirely ignoring others.
Experimental Evaluation
The agents were evaluated across several tabular gridworld environments:
- Low-Impact Environments: Tasks like BreakableBottles and Sokoban require delivering items or reaching goals while minimizing negative side effects (e.g., breaking items or wedging boxes).
- Resource Balancing Environments: A foraging task requiring the agent to balance "Food" and "Drink" collection, plus a new "Gold" resource that does not deplete, testing the agent's ability to maximize a positive objective without compromising safety.
Key Insights and Results
A critical advantage of SFELLA is its continuity. Unlike thresholded methods, SFELLA's smooth curve provides a consistent gradient for the agent to follow, even when far from "safe" zones. This leads to faster learning, as the agent is never "in the dark" about the direction of improvement.
Furthermore, SFELLA addresses the limitations of previous exponential methods like MORE. While MORE becomes indifferent to rewards once a certain level is reached due to its bounded nature, SFELLA’s logarithmic component for gains keeps the agent "hungry" for rewards.
In empirical tests, SFELLA outperformed thresholded methods in 3 out of 4 low-impact tasks and collected significantly more "Gold" in foraging environments compared to MORE. The smoothness of the function was confirmed as the primary driver of performance; "pixelating" the curve into discrete steps led to a drop in efficiency.
Discussion and Future Work
While SFELLA excels in many areas, it struggled in environments like Sokoban, where safety objectives must be perfectly satisfied before any progress is possible. In such cases, rigid thresholds may still be preferable. Additionally, SFELLA assumes rewards are centered around zero; future work could involve agents learning to automatically calibrate their reward scaling to stay within the optimal range of the utility curve. Finally, the non-linear nature of SFELLA could serve as a trigger for human-in-the-loop intervention, identifying moments of "decision paralysis" where AI objectives conflict significantly.
Summary updated