Designing Preferences, Beliefs, and Identities for Artificial Intelligence

Vincent Conitzer

Abstract

In classical artificial intelligence, agents are typically treated as entities with pre-defined identities, preferences, and beliefs. However, as AI systems are increasingly deployed in complex, multi-stakeholder environments, these foundational elements must be reconsidered as design choices. This paper outlines a formal research agenda for architecting AI agents by leveraging social choice theory, game theory, and decision theory. By treating identity, preferences, and beliefs as endogenous variables, we can create systems that are better aligned with human values, privacy requirements, and the logical realities of algorithmic replication.

Introduction

Traditional AI research often views an "agent" as a black box equipped with a clear sense of self, a fixed set of goals, and a consistent worldview. While these assumptions suffice for closed systems like chess or Go, they falter in the "messy" real world. Vincent Conitzer argues that we can no longer take these foundations for granted. Instead, we must learn to design the very DNA of AI—its identity, its preferences, and its belief systems.

The Challenge of Exogenous Foundations

Most AI frameworks assume that an agent’s identity, preferences, and beliefs are exogenously given. Real-world deployment reveals significant cracks in this foundation:

  • Preference Ambiguity: Defining an objective function for systems affecting millions with conflicting values is non-trivial.
  • Identity Blurring: In fleets of identical software instances (e.g., self-driving cars), the boundaries between individual agents become fluid.
  • Logical Dependencies: When multiple AIs run the same code, their actions are logically linked, a phenomenon standard decision theory often fails to capture.

Designing Preferences through Social Choice

A central challenge in AI alignment is determining what a system should want. Asking humans to rank an exponential number of possible outcomes is infeasible. Conitzer proposes eliciting "relative judgments" or trade-offs between specific items or outcomes.

Relative trade-offs between items A, B, and C used to construct a consistent objective function.

By aggregating these pairwise trade-offs (e.g., "How many units of item B is one unit of item A worth?"), we can use social choice theory to derive a linear objective function. This approach allows for the systematic handling of conflicting stakeholder values and ensures the resulting AI policy is mathematically grounded.

Identity as a Design Choice

We often equate an agent with a physical unit, such as a single robot. However, Conitzer suggests defining agents based on information flow. If two nodes do not share data, they function as distinct agents. This perspective allows "identity design" to serve as a tool for privacy and security. By intentionally limiting what an agent remembers or shares—a concept known as "imperfect recall" in game theory—designers can create boundaries that protect user data while maintaining system functionality.

Logical Dependencies and Decision Theory

When two instances of the same AI meet, they are not independent actors but two executions of the same algorithm. Traditional "selfish" game theory may lead to sub-optimal outcomes in these scenarios. The paper explores Functional Decision Theory (FDT) as an alternative. FDT allows agents to recognize that their choices are logically correlated with others running the same code, facilitating more cooperative and efficient behavior.

Furthermore, the paper addresses the "Sleeping Beauty" problem in formal epistemology, exploring how agents should update beliefs when they cannot perfectly locate themselves in time or space. For AI design, the goal is to ensure that the combination of belief and decision theory is immune to "Dutch books" (guaranteed losses).

Conclusion

The design of AI agents requires moving beyond "given" definitions toward a formal architecture of identity and value. By integrating social choice, game theory, and decision theory, researchers can build systems that are not only more capable but also more robustly aligned with the complex social and logical environments they inhabit. Future work must focus on the scalability of these aggregation methods and the development of new formalisms to represent self-locating uncertainty and fluid agent identities.

Summary updated

Abstract

Research in artificial intelligence, as well as in economics and other related fields, generally proceeds from the premise that each agent has a well-defined identity, well-defined preferences over outcomes, and well-defined beliefs about the world. However, as we design AI systems, we in fact need to specify where the boundaries between one agent and another in the system lie, what objective functions these agents aim to maximize, and to some extent even what belief formation processes they use. The premise of this paper is that as AI is being broadly deployed in the world, we need well-founded theories of, and methodologies and algorithms for, how to design preferences, identities, and beliefs. This paper lays out an approach to address these problems from a rigorous foundation in decision theory, game theory, social choice theory, and the algorithmic and computational aspects of these fields.
BibTeX
@inproceedings{conitzer2019designing,
  author = {Vincent Conitzer},
  title = {{Designing Preferences, Beliefs, and Identities for Artificial Intelligence}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33019755},
}
Figure