Introduction
Selective prediction systems are increasingly used in high-stakes domains like healthcare and conservation to manage AI uncertainty. In these systems, an AI model evaluates its own confidence and, when uncertain, defers the decision to a human expert. While much research has focused on optimizing the algorithmic "when to defer" logic, the psychological impact on the human recipient—the "human-in-the-loop"—is often overlooked. This paper investigates how Selective Prediction Messaging (SPM)—the specific information shared with a human during a hand-off—influences team performance.
Methods
The researchers conducted a study using a 2x2 factorial design to test four messaging conditions:
- No Message (NM): The human receives the task with no context.
- Deferral Status Only (DO): The human is informed that the AI has deferred the task.
- Prediction Only (PO): The human sees the AI's tentative prediction but is unaware of the AI's confidence level.
- Both Messages (BM): The human is told the AI deferred and is shown its tentative prediction.
A rule-based deferral mechanism identified "low confidence" instances from an AI model's scores. 198 participants then performed a binary classification task (identifying the presence of animals in images) on these deferred cases.
Dataset: Snapshot Serengeti
The study utilized the Snapshot Serengeti dataset, featuring millions of camera-trap images from Tanzania. The task—identifying animals in often obscured, distant, or false-trigger images—is notoriously difficult for both humans and AI. The researchers selected a subset of "difficult" images where the AI was most uncertain to test human performance under different SPM conditions.
Key Results and Insights
The experimental results demonstrate that the way an AI communicates its uncertainty significantly alters human behavior and accuracy.
The Concentration Effect
The highest human accuracy (61.9%) was achieved in the Deferral Status Only (DO) condition, compared to 58.4% in the baseline No Message (NM) condition. Knowing the AI found the task difficult likely served as a "cognitive forcing function," encouraging participants to examine the images more thoroughly.
The Danger of Anchoring Bias
Conversely, revealing the AI's tentative prediction (PO and BM conditions) degraded performance. In the Prediction Only (PO) condition, accuracy dropped to 57.8%. This suggests an "anchoring bias," where humans tend to conform to the AI's suggestion, even when it is incorrect. This effect was asymmetric: the negative impact of a wrong AI prediction (dropping human accuracy to 41.9%) was far more potent than the positive impact of a correct one.
Conclusion
This research highlights that human-AI team performance is not merely a sum of individual accuracies but is deeply influenced by the interaction interface. By strategically hiding certain information—such as an uncertain AI's tentative guess—designers can "boost" human performance and mitigate harmful anchoring effects. Future work should explore if these findings generalize to expert domains like medical diagnostics and how these interactions evolve over long-term system use.
Summary updated