Reasoned Assumptions and Pareto Optimality

Jon Doyle

Introduction

In the mid-1980s, the field of Artificial Intelligence faced a foundational challenge: how to formalize "common sense" reasoning. Traditional logic is monotonic, meaning that adding new premises never invalidates previous conclusions. However, human reasoning is frequently non-monotonic; we make assumptions (e.g., "birds fly") that we must retract when presented with specific counter-evidence ("this bird is a penguin").

Jon Doyle’s 1985 paper, Reasoned Assumptions and Pareto Optimality, provides a radical departure from the purely logical treatments of this problem. A foundational figure in the development of Truth Maintenance Systems (TMS), Doyle argues that non-monotonic reasoning is less a matter of deduction and more a matter of how an agent chooses its own mental state based on internal preferences. By bridging AI and Economics, Doyle reframes the internal life of an AI agent as a problem of group decision theory.

The Problem: Logic vs. Choice

By the early 1980s, researchers were struggling to provide a solid theoretical foundation for non-monotonic logic. Most treated it as an epistemological problem—a search for what is "true." Doyle suggests this approach is misplaced. He posits that an agent’s mental state is not merely a collection of deduced facts, but a construction built upon "rules of thumb" or desires about what the agent wants to believe to remain functional.

The challenge Doyle tackles is providing a formal justification for these "reasoned assumptions." He moves away from classical logic and toward decision theory, seeking to explain why certain sets of beliefs are more "rational" or "stable" than others.

Methodology: The Mind as a Committee

Doyle introduces a formal structure called a simple reason, represented as:

If I believe A and I don't believe B, then I should believe C.

To evaluate an agent's mental state, he defines two key concepts:

  1. Admissibility: A mental state is admissible if it satisfies all its component "simple reasons" (a closure condition).
  2. Groundedness: A belief is grounded if it has a non-circular argument supporting it, starting from the agent's base observations.

The novel contribution of this paper is the mapping of these AI concepts onto Group Decision Theory. Doyle treats each "reason" within the agent's architecture as an individual "voter" or "economic actor." He then applies the Pareto condition: a state is Pareto optimal if it is impossible to change the state to better satisfy one "reason" without making another "reason" less satisfied.

Key Insights: The Economic Equilibrium of Belief

Doyle’s analysis leads to several provocative conclusions about the nature of artificial and human intelligence:

  • The Society of Mind: Echoing Marvin Minsky, Doyle suggests that an agent is not a monolithic "rational man" (homo economicus). Instead, it is a committee of competing rules and reasons trying to reach a consensus.
  • Belief as Self-Determination: In non-monotonic logic, there are often multiple valid sets of beliefs (known as "extensions"). Doyle argues that choosing between these is an act of self-determination. The agent decides its identity by choosing which reasons to prioritize.
  • The Pareto Equivalence: The paper’s central mathematical proof establishes that the set of beliefs an AI system finds "admissible" is exactly the set that is Pareto optimal. When an AI system reaches a stable state of belief, it has effectively reached an economic equilibrium.

Results and Theoretical Impact

The paper successfully establishes a formal equivalence between "satisfaction optimal" states in a belief system and Pareto optimal sets in a group decision framework. It demonstrates that the "derivability" of a belief—what Doyle calls "validity optimality"—is as much an economic property as a logical one.

Furthermore, Doyle points out that AI systems often violate standard economic axioms, such as the transitivity of preferences. This suggests that AI research can offer economists more realistic models of "rational psychology" that account for the computational and logical limitations inherent in real-world agents.

Limitations and Future Directions

While foundational, the paper focuses on "simple reasons." Doyle acknowledges that a truly comprehensive model of the mind would need to incorporate more complex elements like desires, fears, and intentions, requiring a more sophisticated economic framework. Additionally, in AI systems, the "committee" of reasons is dynamic; the reasons themselves change based on what the agent currently believes. This "reasons about reasons" structure stretches standard economic theory, which typically assumes a fixed set of voters.

Doyle concludes by pointing toward the development of a "rational psychology"—a field that merges AI’s focus on limited computation with economics’ focus on optimal choice. This intersection remains a vital area of study today, particularly in the fields of AI safety and alignment, where understanding how agents resolve internal conflicts is paramount.

Summary updated

BibTeX
@inproceedings{doyle1985reasoned,
  author = {Jon Doyle},
  title = {{Reasoned Assumptions and Pareto Optimality}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1985},
}
Figure