Collective revelation: a mechanism for self-verified, weighted, and truthful predictions

Sharad Goel, Daniel M. Reeves, David M. PennockEC 2009 · DOI

TL;DR

This paper introduces "collective revelation," a novel mechanism that incentivizes experts to provide truthful, self-verified predictions, even when there's no objective "ground truth." It achieves this by cleverly using agents' private information as a proxy for public information, allowing for accurate aggregation of expert opinions weighted by their informativeness.

What Problem Does This Paper Address?

Decision-makers often rely on expert judgment for predictions, especially in areas where objective measurements are difficult or impossible to obtain (e.g., disease prevalence, economic policy impacts). The challenge is to design a system that encourages experts to be honest, accounts for varying levels of expertise, and doesn't require external validation. Existing methods like scoring rules, prediction markets, and peer prediction fall short, failing to simultaneously satisfy all these crucial properties. This paper aims to bridge that gap by proposing a mechanism that achieves all three: incentive compatibility, information weighting, and self-verification.

Methods

The core innovation is the "collective revelation" mechanism. It works by:

  • Incentive Compatibility: Agents are rewarded for truthful reporting.
  • Information Weighting: The mechanism implicitly weights predictions based on the amount of private information an agent possesses. This is achieved by using agents' own private evidence as a proxy for public information.
  • Self-Verification: The system doesn't need external "ground truth" observations. Instead, agents' reports are scored against each other's inferred private information. This creates an equilibrium where truthful revelation is the best strategy.
  • Budget Balancing: A general technique is introduced to ensure that the mechanism makes no net payments to participants, making it economically sustainable. This involves a "shared scoring rule" where agents are rewarded relative to each other, but in a way that prevents them from manipulating their own scores.

The mechanism is formally developed for binary (Bernoulli) outcomes and then extended to normal, Poisson, and exponential distributions, demonstrating its versatility.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Private Information as a Proxy for Public Information: This is the central technical innovation. By using agents' private data to evaluate their reports, the mechanism bypasses the need for objective ground truth.
  • Self-Verification Through Peer Evaluation: Agents are incentivized to be truthful because their reports are judged against the inferred private information of other agents. This creates a robust system where honesty is rewarded.
  • General Budget Balancing Technique: The paper provides a method to ensure that prediction mechanisms are budget-balanced, a crucial practical consideration for real-world implementation. This is achieved by cleverly decoupling scoring from benchmarking.
  • Universality Across Distributions: The mechanism is shown to work for several common types of data distributions (Bernoulli, normal, Poisson, exponential), highlighting its broad applicability.

Key Results

The paper demonstrates that the "collective revelation" mechanism is:

  • Incentive Compatible: Agents are motivated to report truthfully.
  • Information Weighted: Predictions are automatically adjusted based on the informativeness of the expert.
  • Self-Verifying: The mechanism operates without needing objective, external observations.
  • Budget Balanced: The mechanism ensures no net payments are made to participants.

The authors provide formal proofs and theorems showing that these properties are satisfied for various distributions, establishing a strong theoretical foundation for the mechanism.

Limitations and Future Work

  • Common Prior Assumption: The mechanism requires agents to share a common prior belief, which is a strong assumption. The authors suggest that reducing or eliminating this need would be an interesting future direction.
  • Introspection and Communication: The mechanism assumes agents can accurately report their subjective beliefs. In practice, agents might struggle with precise self-assessment or communication. Future work could explore mechanisms that don't require such fine-grained information, perhaps by eliciting relative performance predictions.
  • Generalization to Arbitrary Distributions: While the mechanism works for several common distributions, generalizing it to completely arbitrary distributions is noted as a challenge.

Summary updated

BibTeX
@inproceedings{goel2009collective,
  author = {Sharad Goel and Daniel M. Reeves and David M. Pennock},
  title = {{Collective revelation: a mechanism for self-verified, weighted, and truthful predictions}},
  booktitle = {EC},
  year = {2009},
  doi = {10.1145/1566374.1566413},
}
Figure