TL;DR
This paper investigates whether simple "confidence heuristics" (choosing the more confident or faster judgment) can replace human interaction in collective decision-making. They found that while these heuristics work well for individuals with similar reliability, they perform poorly for dissimilar individuals, highlighting the crucial role of interaction in leveraging diverse expertise.
What Problem Does This Paper Address?
The "two-heads-better-than-one" (2HBT1) effect describes how groups can make more accurate decisions than individuals. A key mechanism for this is sharing "confidence" – an internal estimate of how likely a judgment is correct. This has led to the idea of a "confidence heuristic," where one might simply choose the decision accompanied by higher confidence. Previous work showed this could work even without direct interaction, using algorithms. This paper asks: can these algorithms truly replace interaction, especially when individuals have different levels of expertise (reliability)? They also explore a related heuristic: using reaction time as a proxy for confidence, given their known inverse correlation.
Methods
The researchers designed a perceptual decision-making task where pairs of participants (dyads) had to identify which of two intervals contained a slightly more visible target. After making their private decisions, they shared their confidence levels. The study then compared three strategies for making a collective decision:
- Empirical Dyads: Participants interacted, either non-verbally (just sharing confidence) or verbally (discussing and then deciding).
- Maximum Confidence Slating (MCS): An algorithm that simply picks the decision made with higher confidence. They tested this with both raw and normalized confidence scores.
- Minimum Reaction Time Slating (MRTS): An algorithm that picks the faster decision, leveraging the inverse correlation between confidence and reaction time.
They quantified individual and dyad "sensitivity" (how reliably they made decisions) using psychometric functions and estimated metacognitive accuracy (how well individuals knew their own reliability) using AROC curves.
Datasets
The study analyzed data from an experiment previously conducted by Bahrami et al. (2012a, 2012b). This dataset comprised 58 participants (29 dyads) across two conditions: non-verbal and verbal interaction. No new datasets were introduced.
Key Insights and Ideas
- Similarity is Key for Heuristics: Simple confidence heuristics (like MCS) only outperform interaction when dyad members have very similar levels of reliability.
- Interaction is Superior for Diverse Expertise: When individuals have different levels of reliability, direct interaction is significantly better than algorithmic heuristics. This suggests interaction allows for a more nuanced weighting of opinions.
- Reaction Time is a Poor Substitute for Confidence: The MRTS algorithm, using reaction time as a proxy for confidence, performed worse than the MCS algorithm, even when confidence was negatively correlated with reaction time. This implies reaction time is a noisy and less reliable indicator of judgment quality.
- Metacognition Matters in Interaction: Dyad members actively considered each other's metacognitive accuracy (how well they knew their own reliability) when making joint decisions, especially in disagreement scenarios. This suggests interaction allows for a dynamic assessment of credibility.
- Normalizing Confidence Has Mixed Effects: Normalizing confidence scores (standardizing them across individuals) benefited the MCS algorithm when dyad members were similar but harmed it when they were dissimilar. This points to complex interactions between individual confidence scales and collective decision-making.
Key Results
- The MCS and MRTS algorithms showed collective benefits only when dyad members' sensitivities were highly similar (Smin/Smax > 0.6).
- For dissimilar dyad members (Smin/Smax < 0.6), MCS and MRTS algorithms led to collective losses.
- The MCS algorithm was consistently superior to the MRTS algorithm, regardless of whether raw or normalized confidence estimates were used.
- Interaction (empirical dyads) significantly outperformed the MCS algorithm for dissimilar dyad members (Semp/SMCS > 1 when Smin/Smax < 0.8).
- Dyad members' choice ratios in disagreement trials were predicted by the ratio of their metacognitive accuracies (AROC ratio), indicating they used each other's self-knowledge to guide decisions.
Limitations and Future Work
- Feedback Dependence: The study used feedback, allowing participants to evaluate each other's accuracy. The authors suggest that the benefit of interaction might be reduced in the absence of such diagnostic feedback, though previous work indicates feedback might only accelerate the process.
- Familiarity: Participants were friends, which might influence how they assess credibility. However, prior research suggests familiarity might not be a primary driver of collective performance, with individuals evaluating credibility based on task performance. Future work could explore if familiarity matters more for dissimilar dyads.
- Generalizability to Non-Perceptual Domains: While confidence in perceptual judgments is often "underconfident," knowledge-based judgments can be "overconfident." The authors suggest that findings might generalize due to evidence of a common basis for confidence, but acknowledge this is an open question.
- Mapping Problem: A key challenge for future research is understanding how individuals map internal representations of accuracy (like Ac/σ ratios) onto shared confidence scales, and how to optimize this "mapping problem" for collective decision-making.
- Computational Models: The authors emphasize the need for computational models to better understand how individuals estimate credibility and how social learning models can capture complex decision strategies.
Summary updated