Role of Human-AI Interaction in Selective Prediction

Elizabeth Bondi, Raphael Koster, Hannah Sheahan, Martin J. Chadwick, Yoram Bachrach, A. Taylan Cemgil, Ulrich Paquet, Krishnamurthy DvijothamAAAI 2022 · DOI

Introduction

Selective prediction systems are increasingly used in high-stakes domains like healthcare and conservation to manage AI uncertainty. In these systems, an AI model evaluates its own confidence and, when uncertain, defers the decision to a human expert. While much research has focused on optimizing the algorithmic "when to defer" logic, the psychological impact on the human recipient—the "human-in-the-loop"—is often overlooked. This paper investigates how Selective Prediction Messaging (SPM)—the specific information shared with a human during a hand-off—influences team performance.

The selective prediction pipeline: an AI model processes an input, generates a score, and a deferral model decides whether to use the AI's prediction or defer to a human via SPM.

Methods

The researchers conducted a study using a 2x2 factorial design to test four messaging conditions:

  1. No Message (NM): The human receives the task with no context.
  2. Deferral Status Only (DO): The human is informed that the AI has deferred the task.
  3. Prediction Only (PO): The human sees the AI's tentative prediction but is unaware of the AI's confidence level.
  4. Both Messages (BM): The human is told the AI deferred and is shown its tentative prediction.

A rule-based deferral mechanism identified "low confidence" instances from an AI model's scores. 198 participants then performed a binary classification task (identifying the presence of animals in images) on these deferred cases.

Dataset: Snapshot Serengeti

The study utilized the Snapshot Serengeti dataset, featuring millions of camera-trap images from Tanzania. The task—identifying animals in often obscured, distant, or false-trigger images—is notoriously difficult for both humans and AI. The researchers selected a subset of "difficult" images where the AI was most uncertain to test human performance under different SPM conditions.

Key Results and Insights

The experimental results demonstrate that the way an AI communicates its uncertainty significantly alters human behavior and accuracy.

Human accuracy across the four messaging conditions. The 'Defer only' condition (DO) shows the highest accuracy, while revealing the AI's prediction (PO and BM) leads to lower performance.

The Concentration Effect

The highest human accuracy (61.9%) was achieved in the Deferral Status Only (DO) condition, compared to 58.4% in the baseline No Message (NM) condition. Knowing the AI found the task difficult likely served as a "cognitive forcing function," encouraging participants to examine the images more thoroughly.

The Danger of Anchoring Bias

Conversely, revealing the AI's tentative prediction (PO and BM conditions) degraded performance. In the Prediction Only (PO) condition, accuracy dropped to 57.8%. This suggests an "anchoring bias," where humans tend to conform to the AI's suggestion, even when it is incorrect. This effect was asymmetric: the negative impact of a wrong AI prediction (dropping human accuracy to 41.9%) was far more potent than the positive impact of a correct one.

Conclusion

This research highlights that human-AI team performance is not merely a sum of individual accuracies but is deeply influenced by the interaction interface. By strategically hiding certain information—such as an uncertain AI's tentative guess—designers can "boost" human performance and mitigate harmful anchoring effects. Future work should explore if these findings generalize to expert domains like medical diagnostics and how these interactions evolve over long-term system use.

Summary updated

BibTeX
@inproceedings{bondi2022role,
  author = {Elizabeth Bondi and Raphael Koster and Hannah Sheahan and Martin J. Chadwick and Yoram Bachrach and A. Taylan Cemgil and Ulrich Paquet and Krishnamurthy Dvijotham},
  title = {{Role of Human-AI Interaction in Selective Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i5.20465},
}
Figure