Abstract
Autonomous systems can substantially enhance a human's efficiency and effectiveness in complex environments. Machines, however, are often unable to observe the preferences of the humans that they serve. Despite the fact that the human's and machine's objectives are aligned, asymmetric information, along with heterogeneous sensitivities to risk by the human and machine, make their joint optimization process a game with strategic interactions. We propose a framework based on risk-sensitive dynamic games; the human seeks to optimize her risk-sensitive criterion according to her true preferences, while the machine seeks to adaptively learn the human's preferences and at the same time provide a good service to the human. We develop a class of performance measures for the proposed framework based on the concept of regret. We then evaluate their dependence on the risk-sensitivity and the degree of uncertainty. We present applications of our framework to self-driving taxis, and robo-financial advising.
Introduction
As autonomous systems become increasingly integrated into high-stakes environments—ranging from autonomous vehicles to financial management—the challenge of "value alignment" becomes paramount. Most current autonomous systems are designed under a risk-neutral paradigm, aiming to minimize the expected cost of a task. However, human decision-makers are rarely risk-neutral. Individual preferences regarding uncertainty vary significantly; for instance, one commuter might prioritize a predictable 30-minute route, while another might risk a 20-minute route that has a high probability of traffic delays.
This research addresses two fundamental hurdles in human-machine cooperation:
- Informational Asymmetry: The machine cannot directly observe the human's internal risk tolerance (their "type").
- Strategic Interaction: Even when objectives are nominally aligned, the machine’s uncertainty about the human’s preferences transforms the interaction into a strategic game where every action serves as a signal.
While existing frameworks like Cooperative Inverse Reinforcement Learning (CIRL) address preference learning, they typically assume risk-neutrality. This paper proposes a risk-sensitive dynamic game framework to bridge this gap.
Methodology
Risk-Sensitive Bayesian Equilibria
The proposed framework moves beyond standard Nash equilibria to account for two distinct types of risk: context-driven risk (environmental uncertainty) and human-driven risk (uncertainty regarding the human's risk-sensitivity parameter, $\theta$). By modeling the interaction as a risk-sensitive dynamic game with asymmetric information, the system can account for the "downside risk" that is often critical to human users.
The Coordinator Technique
To address the mathematical complexity of multi-agent strategic interactions, the authors employ a "coordinator technique." This method reduces the game to a single-agent optimization problem by introducing a virtual coordinator that selects strategies for all possible human types. This transformation allows the problem to be solved as a risk-sensitive Partially Observable Markov Decision Process (risk-POMDP).
Convex Risk Measures and Bayesian Learning
Instead of simple expected utility, the framework utilizes convex risk measures (such as mean-variance) to quantify how agents weight adverse outcomes. The machine maintains a belief—a probability distribution—over the human's risk parameters. As the human interacts with or overrides the machine, the system applies Bayes' rule to update this belief, effectively learning the human's "type" in real-time.
Experimental Evaluation
The framework was evaluated through a numerical study involving a stochastic shortest path problem. The environment consisted of a grid where each edge was assigned a random cost characterized by a specific mean and variance. The simulation modeled a human with a hidden risk parameter $\theta$ interacting with a self-driving taxi.
The study compared three primary strategies:
- Human-Machine (Proposed): The machine adaptively learns the human's risk preferences.
- Machine-Neutral: The machine ignores variance and only minimizes expected costs.
- Machine-Average: The machine targets a "typical" or average human risk profile.
Performance was measured using Regret, defined as the difference between the actual outcome and the ideal outcome achieved if the machine had perfect knowledge of the human's preferences from the start.
Key Insights and Results
Actions as High-Signal Data
A critical finding is that human interventions—such as overriding an autonomous system—are not merely corrections but are highly informative signals. In many cases, a single override was "fully revealing," allowing the machine to perfectly identify the human's risk parameter and optimize all subsequent actions.
The Failure of Risk-Neutrality
The "Machine-Neutral" strategy performed the worst, particularly when paired with highly risk-averse humans. This highlights the "cost of neutrality": if a machine ignores the human's sensitivity to variance, the resulting regret spikes, leading to a significant breakdown in cooperation and trust.
Superiority of Personalization
The "Human-Machine" strategy consistently achieved the lowest regret across all scenarios. By actively minimizing the human's risk-adjusted cost, the system demonstrated that true value alignment requires the machine to predict how a specific individual values uncertain outcomes rather than relying on a population average.
Limitations and Future Directions
While the framework provides a robust foundation for risk-sensitive cooperation, several challenges remain:
- Time-Separability: Many sophisticated risk measures do not satisfy the Markov property (time-separability), meaning the risk of a total journey cannot be easily decomposed into individual steps. This complicates the recursive optimization required for long-term tasks.
- Computational Complexity: Solving POMDPs is PSPACE-complete. Scaling this framework from grid-world simulations to complex, real-world environments requires significant engineering and algorithmic refinement.
- Subgame Perfection: Future iterations must ensure "incentive compatibility" at every stage of the interaction, ensuring that neither the human nor the machine has a strategic reason to deviate from the cooperative path at any point in time.
Summary updated