TL;DR
This paper introduces a novel method for aggregating expert opinions that allows experts to dynamically adjust their trust in each other based on how closely their opinions align. This "consensual" approach leads to a single, unified opinion and is shown to be more accurate than traditional averaging methods, especially when dealing with a large number of experts.
What Problem Does This Paper Address?
When we need to make predictions about uncertain events, especially when historical data is scarce, we often turn to experts. The challenge then becomes how to combine the diverse opinions of these experts into a single, reliable forecast. Traditional methods like simple averaging can be easily swayed by a single outlier opinion. More sophisticated methods exist, but they often struggle with how to dynamically adjust the influence of each expert over time or how to ensure experts are incentivized to be truthful. This paper tackles the problem of creating a robust and adaptive opinion aggregation system that encourages experts to converge on a consensus while naturally incorporating their evolving trust in one another.
Methods
The core of the proposed method is an iterative process where experts continuously update their opinions. Each expert's updated opinion is a linear combination of all experts' current opinions, but crucially, the weights assigned to each peer's opinion are not fixed. Instead, they are inversely proportional to the "distance" between the expert's own opinion and the peer's opinion. This means experts naturally give more weight to opinions that are closer to their own. The paper defines this distance using a root-mean-square deviation.
The authors prove that this iterative updating process leads to a consensus, where all experts' opinions converge to a single, shared opinion. They also demonstrate that if experts are incentivized using a quadratic scoring rule (a common method for rewarding accurate predictions), the assumption that they prefer opinions close to their own naturally emerges. This provides a theoretical justification for their weighting scheme. The algorithm is described as a matrix operation where the weight matrix is updated at each step based on the current opinions.
Datasets
The paper evaluates its proposed method using a dataset of 267 National Football League (NFL) games from the 2005-2006 season. The opinions were collected from the ProbabilityFootball contest, where 519 experts provided their subjective probabilities for each team winning a game. On average, about 432 experts contributed predictions per game. The contest also used a reward system based on a positive affine transformation of the quadratic scoring rule, which aligns with the theoretical underpinnings of the proposed method.
Key Insights and Ideas
- Dynamic Trust: The most significant insight is the idea of experts dynamically adjusting their trust in each other based on opinion similarity. This moves beyond static weighting schemes and allows for a more nuanced aggregation.
- Inherent Consensus Mechanism: The method's design inherently drives experts towards a consensus, meaning the process doesn't require an external arbiter to force agreement.
- Theoretical Justification for Weighting: Linking the weighting scheme to the quadratic scoring rule provides a strong theoretical foundation, suggesting that rational, incentivized experts would naturally adopt this behavior.
- Robustness to Outliers: By down-weighting distant opinions, the method is less susceptible to being skewed by extreme or erroneous predictions from a few experts, especially in large groups.
Key Results
The experimental evaluation on NFL game predictions showed that the proposed "consensual" method outperformed two baseline methods: a traditional average approach and the Barlow et al. (BMS) method.
- Overall Accuracy: The consensual method achieved the best overall accuracy at 69.29%, compared to 68.54% for BMS and 67.42% for the average method.
- Absolute Error: The consensual method also had the lowest average absolute error (0.4115), significantly outperforming the average method (0.4176) and the BMS method (0.4295). Statistical tests confirmed the significance of these results.
- The paper highlights that the BMS method, which only considers the distance to the most distant opinion, performed poorly in absolute error, likely due to its sensitivity to extreme opinions. The average method, while seemingly robust, was found to overweight extreme opinions when all experts were equally weighted.
Limitations and Future Work
- Computational Cost: While the method leads to consensus, the iterative nature of updating opinions and weights might be computationally intensive, especially with a very large number of experts and many iterations. The paper presents an algorithmic description but doesn't deeply analyze its computational complexity.
- Definition of Distance: The paper uses a specific root-mean-square deviation for measuring opinion distance. Exploring other distance metrics and their impact on the consensus process could be a future direction.
- Expert Rationality Assumption: The theoretical justification relies on experts being rational and maximizing their expected scores. Real-world experts might have other motivations or biases not fully captured by this model.
- Future Work: The authors suggest exploring how to derive weights from reported opinions to preserve privacy, as experts don't need to reveal their opinions to others to assign weights. They also hint at the potential for this method in various decision-making scenarios.
Summary updated