Decision Making without Common Priors

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 239 papers43 conceptsfrozen 2026-10-04

In classical game theory, there is a ghost in the machine: the Common Prior Assumption (CPA). It posits that if rational agents had access to the same information, they would necessarily hold the same beliefs about the world. This assumption is the bedrock of Aumann’s famous "Agreement Theorem," which proves that honest, rational agents can never "agree to disagree."

However, as Scott Aaronson notes in The Complexity of Agreement, experts in the real world disagree constantly. This suggests that the barrier to consensus isn't just communication or computation—it is the fact that we do not start with the same "prior" probability distributions. Research into Heterogeneous Priors seeks to rebuild mechanism design and decision theory for a world where agents fundamentally disagree on how the world works.

The Fragility of the Common Prior

Most traditional mechanisms for eliciting information—like scoring rules used in weather forecasting—assume the principal and the agents share a model of the world. Aviv Zohar and Jeffrey S. Rosenschein argue in Robust Mechanisms for Information Elicitation that when agents possess divergent beliefs due to different data sources, these standard incentive structures break down.

If I believe a coin is 60% likely to be heads and you believe it is 40%, a mechanism that assumes we both think it's 50% will fail to elicit our true private signals. This mismatch is not a "failure" of rationality. As Alan Jern and Kai-min Chang demonstrate in Bayesian Belief Polarization, even extreme belief polarization can be a mathematically sound outcome of Bayesian inference if agents start with different, internally consistent causal models.

Eliciting Truth from Subjective Peers

How do you reward someone for an honest review of a movie or a product when there is no "ground truth" to verify it against? This is the domain of Peer Prediction.

Early peer-prediction models relied heavily on the CPA, assuming everyone updated their beliefs in the exact same way. Goran Radanovic and Boi Faltings have pioneered mechanisms that relax this. In Incentives for Subjective Evaluations with Private Beliefs, they introduce frameworks that elicit truthful reports even when agents have diverse backgrounds and private beliefs.

A key breakthrough in this area is the Peer Truth Serum (PTS), detailed in Peer Truth Serum: Incentives for Crowdsourcing Measurements and Opinions. Instead of just rewarding agreement, PTS rewards agents more for agreeing on uncommon reports. This forces the system to converge toward the true distribution of a phenomenon without ever needing to know the "correct" answer beforehand. Grant Schoenebeck and Yuqing Kong further strengthened this in Equilibrium Selection in Information Elicitation without Verification via Information Monotonicity, proving that truth-telling can be made the most profitable strategy even when agents try to collude.

Aggregation as a Bet-Settling Device

When a group must make a collective decision—like a jury or an AI serving multiple users—divergent priors create a "social choice" nightmare. Standard methods of averaging opinions fail because they can't distinguish between an agent's private information and their initial bias.

Nishant Desai proposes a dynamic solution in Negotiable Reinforcement Learning for Pareto Optimal Sequential Decision-Making. Rather than a static compromise, the system acts as a "bet-settling" device: it gradually shifts its priority toward whichever stakeholder’s predictions are being vindicated by real-world data.

Surprisingly, "incorrect" beliefs can sometimes be a feature rather than a bug. Daewon Seo and Ravi Kiran Raman show in Decision Making in Star Networks with Incorrect Beliefs that in certain network structures, agents with biased or "wrong" priors can actually help the central aggregator reach a more accurate collective decision by counteracting other systemic noises.

The Frontier: AI Alignment

The stakes for this research are highest in AI Alignment. As Aran Nayebi argues in Barriers and Pathways to Human-AI Alignment: A Game-Theoretic Approach, the fundamental limit of alignment is that humans and AI systems start with divergent world models and internal representations.

If we assume a common prior, alignment looks like a simple optimization problem. But if we accept that the human and the AI operate with fundamentally different priors, alignment becomes an information-theoretic challenge of bridging two subjective worlds. Moving beyond the common prior assumption is not just about making better prediction markets; it is about building systems that can reason alongside humans who see the world differently.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 27 papers
  2. Version 1 · 2026-10-04 · 39 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_common_prior_assumption_807f6db7,
  title = {common prior assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/common-prior-assumption/rdr_807f6db77abc4969bb3a9e9eaa54333e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_decision_making_without_common_priors_807f6db7_v1,
  title = {Decision Making without Common Priors},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decision-making-without-common-priors/rdr_807f6db77abc4969bb3a9e9eaa54333e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}