Using soft maximin for risk averse multi-objective decision-making

Benjamin J. Smith, Robert Klassert, Roland PihlakasAutonomous 2022 · DOI

Abstract

Balancing multiple competing and conflicting objectives is an essential task for any artificial intelligence tasked with satisfying human values or preferences. Conflict arises both from misalignment between individuals with competing values, but also between conflicting value systems held by a single human. Starting with the principle of loss-aversion, we designed a set of soft maximin function approaches to multi-objective decision-making. Benchmarking these functions in a set of previously-developed environments, we found that one new approach in particular, ‘split-function exp-log loss aversion’ (SFELLA), learns faster than the state-of-the-art thresholded alignment objective method on three of four tasks it was tested on, and achieved the same optimal performance after learning. SFELLA also showed relative robustness improvements against changes in objective scale, which may highlight an advantage dealing with distribution shifts in the environment dynamics. Overall, we found SFELLA useful for avoiding problems that sometimes occur with a thresholded approach, and more reward-responsive than previous exponential methods while retaining its conservative, loss-averse incentive structure.

Introduction

In AI Safety, "alignment" typically involves ensuring an agent follows human values. However, human values are rarely represented by a single scalar; they are often a complex, conflicting set of multiple objectives (e.g., "get the coffee" vs. "don't knock over the vase"). Existing approaches frequently rely on rigid safety thresholds. While intuitive, these thresholds create significant challenges in Reinforcement Learning (RL), such as gradient sparsity—where an agent receives no feedback until a threshold is crossed—and decision paralysis, where an agent avoids all movement to minimize any risk of threshold violation. This research seeks a balance that is loss-averse (prioritizing the avoidance of bad outcomes) yet reward-responsive (willing to pursue gains once safety is reasonably assured).

Methodology: The SFELLA Approach

The proposed solution is a "soft maximin" scalarization. Instead of summing raw rewards, the agent transforms each objective's reward through a non-linear utility function $f(x)$ before summation. The primary contribution is SFELLA (Split-Function Exp-Log Loss Aversion).

SFELLA is characterized by a split utility curve:

  • For Negative Rewards (Losses): It employs an exponential decay. This ensures the "pain" of a loss grows rapidly, compelling the agent to prioritize the worst-performing objective.
  • For Positive Rewards (Gains): It utilizes a logarithmic function. This provides diminishing returns for gains but remains unbounded, ensuring the agent continues to seek improvements.
Comparison of utility functions for different multi-objective approaches. SFELLA (yellow) combines exponential loss aversion with logarithmic gain utility.

By summing these transformed values, the agent naturally behaves like a soft maximin operator, focusing attention on the lowest-performing objective without entirely ignoring others.

Experimental Evaluation

The agents were evaluated across several tabular gridworld environments:

  1. Low-Impact Environments: Tasks like BreakableBottles and Sokoban require delivering items or reaching goals while minimizing negative side effects (e.g., breaking items or wedging boxes).
  2. Resource Balancing Environments: A foraging task requiring the agent to balance "Food" and "Drink" collection, plus a new "Gold" resource that does not deplete, testing the agent's ability to maximize a positive objective without compromising safety.

Key Insights and Results

A critical advantage of SFELLA is its continuity. Unlike thresholded methods, SFELLA's smooth curve provides a consistent gradient for the agent to follow, even when far from "safe" zones. This leads to faster learning, as the agent is never "in the dark" about the direction of improvement.

Furthermore, SFELLA addresses the limitations of previous exponential methods like MORE. While MORE becomes indifferent to rewards once a certain level is reached due to its bounded nature, SFELLA’s logarithmic component for gains keeps the agent "hungry" for rewards.

Responsiveness (gradient) of utility functions. SFELLA remains more responsive to positive rewards compared to the MORE approach, which flattens out.

In empirical tests, SFELLA outperformed thresholded methods in 3 out of 4 low-impact tasks and collected significantly more "Gold" in foraging environments compared to MORE. The smoothness of the function was confirmed as the primary driver of performance; "pixelating" the curve into discrete steps led to a drop in efficiency.

Discussion and Future Work

While SFELLA excels in many areas, it struggled in environments like Sokoban, where safety objectives must be perfectly satisfied before any progress is possible. In such cases, rigid thresholds may still be preferable. Additionally, SFELLA assumes rewards are centered around zero; future work could involve agents learning to automatically calibrate their reward scaling to stay within the optimal range of the utility curve. Finally, the non-linear nature of SFELLA could serve as a trigger for human-in-the-loop intervention, identifying moments of "decision paralysis" where AI objectives conflict significantly.

Summary updated

Abstract

Abstract Balancing multiple competing and conflicting objectives is an essential task for any artificial intelligence tasked with satisfying human values or preferences. Conflict arises both from misalignment between individuals with competing values, but also between conflicting value systems held by a single human. Starting with principle of loss-aversion, we designed a set of soft maximin function approaches to multi-objective decision-making. Bench-marking these functions in a set of previously-developed environments, we found that one new approach in particular, ‘split-function exp-log loss aversion’ (SFELLA), learns faster than the state of the art thresholded alignment objective method Vamplew (Engineering Applications of Artificial Intelligenceg 100:104186, 2021) on three of four tasks it was tested on, and achieved the same optimal performance after learning. SFELLA also showed relative robustness improvements against changes in objective scale, which may highlight an advantage dealing with distribution shifts in the environment dynamics. We further compared SFELLA to the multi-objective reward exponentials (MORE) approach, and found that SFELLA performs similarly to MORE in a simple previously-described foraging task, but in a modified foraging environment with a new resource that was not depleted as the agent worked, SFELLA collected more of the new resource with very little cost incurred in terms of the old resource. Overall, we found SFELLA useful for avoiding problems that sometimes occur with a thresholded approach, and more reward-responsive than MORE while retaining its conservative, loss-averse incentive structure.
BibTeX
@inproceedings{smith2022using,
  author = {Benjamin J. Smith and Robert Klassert and Roland Pihlakas},
  title = {{Using soft maximin for risk averse multi-objective decision-making}},
  booktitle = {Autonomous},
  year = {2022},
  doi = {10.1007/s10458-022-09586-2},
}
Figure