Effect of Artificial Intelligence Helpfulness and Uncertainty on Cognitive Interactions with Pharmacists: Randomized Controlled Trial

Chuan-Ching Tsai, Jin Yong Kim, Qiyuan Chen, Brigid Rowell, X. Jessie Yang, Raed Al Kontar, Megan Whitaker, Corey A. LesterJournal of Medical Internet Research 2024 · DOI

TL;DR

This study investigated how pharmacists interact with AI-powered medication verification tools, specifically examining the impact of AI helpfulness and uncertainty communication on their cognitive processes. The findings reveal that while AI can shift pharmacists' attention, displaying AI uncertainty can increase cognitive load, suggesting a need for user-centered design to optimize AI integration in clinical workflows.

What Problem Does This Paper Address?

The increasing integration of Artificial Intelligence (AI) into healthcare, particularly in clinical decision support systems (CDSS), presents a challenge in understanding how these tools affect human cognition and interaction. Specifically, for pharmacists performing medication verification – a critical yet time-consuming task prone to errors – it's unclear how AI assistance, its accuracy (helpfulness), and the communication of its uncertainty influence their cognitive workload and decision-making. While AI promises to reduce errors and cognitive load, poorly designed interfaces or misleading information could inadvertently increase these issues, potentially leading to automation bias or algorithmic aversion. This paper aims to bridge this knowledge gap by analyzing pharmacists' cognitive interactions with an AI prototype during medication verification.

Methods

The researchers conducted a randomized controlled trial involving 30 pharmacists. Participants performed 200 medication verification tasks each, with their eye movements recorded using an online eye tracker. These tasks were divided into two conditions: 100 verifications without AI assistance and 100 with AI assistance. The AI assistance was further categorized into two types: "black box" (providing AI predictions without uncertainty information) and "uncertainty-aware" (displaying AI predictions along with a probability histogram indicating uncertainty).

The core of the analysis involved examining eye-tracking data, specifically fixation patterns. This included:

  • First and last fixation regions: Where participants initially and finally focused their attention.
  • Number of fixations: How many times participants looked at specific areas.
  • Fixation duration and dwell times: How long participants spent looking at specific areas, indicative of cognitive processing.

The AI software itself was developed using a Bayesian neural network to predict medication characteristics (NDC, color, shape, imprint) and their associated uncertainty. The model was intentionally oversampled for errors to ensure a sufficient number of "unhelpful" AI instances for analysis.

Datasets

Not explicitly discussed in the paper as distinct datasets for training or evaluation beyond the data generated by the study participants themselves during the experimental tasks. The AI model was trained on prescription images.

Key Insights and Ideas

  • AI shifts attention, but doesn't eliminate core tasks: Pharmacists significantly shifted their visual attention (19%-26% of fixations) to AI-generated regions when available, indicating integration of AI advice. However, the "fill image" (the actual medication to be verified) remained the primary focus area, suggesting AI acts as a supplementary tool rather than a replacement for direct inspection.
  • Unhelpful AI increases cognitive burden: When the AI provided incorrect advice ("unhelpful"), pharmacists spent longer dwelling on the reference and fill images. This suggests increased cognitive processing, likely due to confusion or the need to re-verify information.
  • Uncertainty communication adds cognitive load: Displaying AI uncertainty (via a histogram) led to longer cognitive processing times, as measured by dwell times in original images. This implies that while transparency is important, presenting uncertainty can also be cognitively demanding, potentially causing users to second-guess the AI or spend more time reconciling the information.
  • "Black box" AI might lead to overreliance: Shorter dwell times with "black box" AI advice, especially when it was unhelpful, could indicate a tendency towards automation bias, where users might accept incorrect AI suggestions without sufficient scrutiny.
  • User-centered design is paramount: The study underscores the need for AI systems to be designed with the end-user in mind, balancing the benefits of transparency with the potential for information overload.

Key Results

  • Fixation shifts: Pharmacists allocated 19%-26% of their total fixations to AI-generated regions when AI was available.
  • Cognitive processing with unhelpful AI: Unhelpful AI advice led to longer dwell times on reference and fill images, indicating increased cognitive processing.
  • Cognitive processing with uncertainty: Displaying AI uncertainty led to longer cognitive processing times, measured by dwell times in original images.
  • Accuracy matters: Helpful AI advice resulted in shorter dwell times in reference and fill images compared to unhelpful advice.
  • No reduction in fill image fixations: AI assistance did not reduce the number of fixations on fill images, which remained the primary focus.

Limitations and Future Work

Limitations: * Limited eye-tracking metrics: The study acknowledges that other eye-tracking metrics (e.g., saccade length, pupil dilation) could provide further insights into cognitive load but were not included due to data availability. * AI accuracy manipulation: The AI's accuracy was intentionally lowered to oversample errors, which might not perfectly reflect real-world AI performance.

Future Work: * Explore additional eye-tracking metrics: To gain a more comprehensive understanding of cognitive workload. * User-centered design and usability testing: Conduct more extensive user-centered design and usability studies to inform CDSS development. * Qualitative and quantitative methods: Integrate interviews and surveys to better understand user experiences and perceptions. * Customizable AI displays: Allow users to customize AI displays based on their expertise and comfort level. * Broader application: Investigate AI integration in other dispensing tasks beyond verification to reduce overall cognitive load.

Summary updated

Abstract

ClinicalTrials.gov NCT06795477; https://clinicaltrials.gov/study/NCT06795477.
BibTeX
@article{tsai2024effect,
  author = {Chuan-Ching Tsai and Jin Yong Kim and Qiyuan Chen and Brigid Rowell and X. Jessie Yang and Raed Al Kontar and Megan Whitaker and Corey A. Lester},
  title = {{Effect of Artificial Intelligence Helpfulness and Uncertainty on Cognitive Interactions with Pharmacists: Randomized Controlled Trial}},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/59946},
}
Figure