Servant of Many Masters: Shifting priorities in Pareto-optimal sequential decision-making

Andrew Critch, Stuart RussellarXiv 2017 · OpenReview

TL;DR

This paper tackles the challenge of designing AI agents that serve multiple principals with potentially conflicting beliefs and preferences. It demonstrates that traditional methods of aggregating preferences (like Harsanyi's theorem) fail when principals have different prior beliefs. The authors introduce a novel framework where an agent's prioritization of principals dynamically shifts based on how well incoming observations align with each principal's beliefs, effectively creating a "bet-settling" mechanism.

What Problem Does This Paper Address?

As AI systems become more integrated into our lives, making decisions on behalf of multiple stakeholders (e.g., companies, governments, individuals) is becoming increasingly important. A natural goal is for such an agent to be "Pareto-optimal," meaning it cannot improve one principal's outcome without worsening another's. Harsanyi's influential theorem suggests that a Pareto-optimal policy can be achieved by a fixed weighted sum of each principal's utility. However, this theorem assumes principals share common beliefs about the world. This paper addresses the crucial gap: what happens when principals have different prior beliefs about the environment, and how should an agent navigate this to remain Pareto-optimal, especially in sequential decision-making scenarios where new information arrives over time?

Methods

The core of the paper lies in extending Harsanyi's social aggregation theorem to the case of differing principal beliefs within a Partially Observable Markov Decision Process (POMDP) framework. The authors introduce the concept of a "POMDP mixture" to represent a single decision problem that combines the perspectives of multiple principals.

Key methodological contributions include:

  • Generalization of Harsanyi's Theorem: The paper shows that Harsanyi's theorem does not hold when priors differ. They derive a more precise generalization.
  • Dynamic Prioritization: The main result is a theorem (Theorem 3) that characterizes Pareto-optimal policies. This theorem reveals that the agent's prioritization of principals is not fixed but evolves dynamically. The agent adjusts its focus based on how well incoming observations confirm or contradict each principal's prior beliefs. This is described as a "bet-settling" mechanism.
  • Geometric and Probabilistic Characterization: They provide both geometric (Polytope Lemma) and probabilistic characterizations of Pareto-optimality, linking it to optimal policies in POMDP mixtures.
  • Backward Recursion: A key result is a backward recursive formula for determining optimal policies, similar to Bellman's principle of optimality but adapted for multiple principals with differing beliefs.

Datasets

Not discussed in the paper. The paper focuses on theoretical development and uses a hypothetical "cake betting" scenario for illustration.

Key Insights and Ideas

  • Beliefs Matter More Than Just Utilities: The paper powerfully demonstrates that differences in beliefs are as critical as differences in utility functions when designing multi-principal AI systems. Ignoring these belief differences leads to suboptimal Pareto-optimal policies.
  • Dynamic "Bet-Settling": The most striking insight is that Pareto-optimality in the presence of differing beliefs inherently involves a form of "bet-settling." The agent implicitly favors principals whose beliefs are better supported by incoming evidence, allowing them to "win" their bets and gain more utility. This is a departure from static, fixed-weight aggregation.
  • Pareto-Optimality as a Geometric Property: The Polytope Lemma provides an elegant geometric interpretation of Pareto-optimality, showing it relates to the convex hull of achievable utility outcomes.
  • Implications for Real-World Contracts and AI Adoption: The findings have direct implications for designing contracts, treaties, and joint ventures. They also suggest that AI systems designed with this dynamic Pareto-optimality in mind are more likely to be adopted for shared ownership, as they can better accommodate diverse stakeholder interests.

Key Results

  • Theorem 3: This is the central result, providing a backward recursive formula for Pareto-optimal policies for an agent serving two principals with different beliefs. It shows that the agent must use each principal's own world-model and dynamically shift priorities based on how well observations align with each principal's prior.
  • Proposition 1: Demonstrates that a Pareto-optimal policy that accounts for differing beliefs (like the "cake betting" example) cannot be implemented by simply maximizing a fixed linear combination of utilities. Such a policy is strictly worse for at least one principal compared to the dynamic approach.
  • Corollary 4: Shows that if principals share the same beliefs, the derived Pareto-optimal policy reduces to Harsanyi's classical result of a fixed weighted sum of utilities.

Limitations and Future Work

  • Common Knowledge Assumption: The paper assumes principals have common knowledge of each other's priors at the time of agent creation. The authors acknowledge this is a strong assumption and suggest future work on how principals might reach such an equilibrium or how to obviate the need for it.
  • Technical Sophistication: While the core theorem is presented as not overly mathematically complex, implementing these dynamic policies in real-world AI systems remains a significant technical challenge.
  • Future Work Directions:
    • Developing methods for principals to share information or adjust weights to reach agreement.
  • Modeling bargaining scenarios with greater transparency between principals.
  • Accounting for principals who continue to exert active control over the system after its creation.
  • Exploring "machine implementable social choice theory" to bridge the gap between theoretical results and practical implementation.

Summary updated

Abstract

It is often argued that an agent making decisions on behalf of two or more principals who have different utility functions should adopt a {\em Pareto-optimal} policy, i.e., a policy that cannot be improved upon for one agent without making sacrifices for another. A famous theorem of Harsanyi shows that, when the principals have a common prior on the outcome distributions of all policies, a Pareto-optimal policy for the agent is one that maximizes a fixed, weighted linear combination of the principals' utilities. In this paper, we show that Harsanyi's theorem does not hold for principals with different priors, and derive a more precise generalization which does hold, which constitutes our main result. In this more general case, the relative weight given to each principal's utility should evolve over time according to how well the agent's observations conform with that principal's prior. The result has implications for the design of contracts, treaties, joint ventures, and robots.
BibTeX
@misc{critch2017servant,
  author = {Andrew Critch and Stuart Russell},
  title = {{Servant of Many Masters: Shifting priorities in Pareto-optimal sequential decision-making}},
  year = {2017},
  eprint = {1711.00363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.00363},
}

Loading related content...

Figure