TL;DR
This study investigated patient confidence in AI, specifically large language models (LLMs), within a clinical urology setting. After interacting with a GPT-4 powered chatbot, patients found LLMs valuable for extending consultation time but still preferred human urologists for critical tasks like diagnosis and treatment recommendations, suggesting AI's role as a complement rather than a replacement.
What Problem Does This Paper Address?
The integration of Artificial Intelligence (AI), particularly Large Language Models (LLMs), into healthcare is rapidly advancing. However, there's a significant gap in understanding how patients perceive and trust AI in clinical settings, especially after direct interaction. Previous studies often relied on surveys without hands-on experience, potentially introducing biases due to a lack of familiarity with AI. This paper aims to address this by providing real-world urological patients with a controlled interaction with an LLM-powered chatbot to gauge their confidence in its clinical capabilities compared to human urologists.
Methods
The study employed a prospective, interventional design with a pre- and post-interaction survey. 300 eligible patients scheduled for urological counseling were enrolled.
- Pre-interventional Phase: Participants received an introduction to LLMs and completed a survey assessing their information-seeking preferences, baseline knowledge, and affinity for AI.
- Interventional Phase: Patients interacted with a GPT-4 powered chatbot on a tablet, using speech-to-text and text-to-speech functionalities to ask medical questions relevant to their current clinical situation. The chatbot was pre-prompted with keywords related to common urological concerns. Study personnel were available to correct any misinformation.
- Post-interventional Phase: Participants completed a survey evaluating their confidence in AI versus physicians across various clinical capabilities, alongside a demographic questionnaire.
Confidence levels were assessed using a 0-5 Likert scale. Statistical analysis included paired sample t-tests to compare confidence levels and multiple linear regression to identify predictors of AI confidence.
Datasets
Not discussed in the paper. The study utilized a custom-designed survey and a GPT-4 powered chatbot for the intervention.
Key Insights and Ideas
- AI as a Complement, Not a Replacement: The most striking finding is the clear patient preference for AI as a supportive tool rather than a substitute for human expertise in core clinical decision-making.
- Time is of the Essence: Patients highly valued the extended consultation time offered by the AI, highlighting a critical unmet need in healthcare delivery.
- Age is Not a Barrier: The study found no significant correlation between age and confidence in AI, suggesting LLMs are user-friendly and accessible across diverse age groups, which is particularly relevant for the often older demographic in urology.
- Trust in Online Sources Matters: Confidence in AI was significantly predicted by a patient's existing trust in online information sources, underscoring the importance of digital literacy and established trust mechanisms.
Key Results
- Preference for Urologists: Urologists were overwhelmingly favored for history taking (mean difference 0.78), diagnostics (1.02), treatment recommendations (1.05), guideline concordance (0.62), and anxiety reduction (1.60).
- AI Wins on Time: AI was significantly preferred for allocated consultation time (mean difference -0.59), with patients reporting higher confidence in AI's ability to provide sufficient time.
- High Engagement: Out of 300 enrolled patients, 292 (97.3%) completed the study.
- Limited Prior AI Experience: A significant majority (89%) of participants had never used LLMs for medical question answering (medQA).
- Google Dominance: Most patients (82%) used Google as their primary source for medical information.
- Predictors of Confidence: Only confidence in online information sources was a significant predictor of AI confidence (β = 0.72, p = 0.018). Age did not predict confidence.
Limitations and Future Work
- Social Desirability Bias: The authors acknowledge the potential for social desirability bias in direct questioning. They plan to address this in future studies using indirect questioning methods.
- Sampling Bias: The study sample was predominantly male and elderly, which, while representative of typical urological patient populations, might limit generalizability to other demographics or specialties.
- Future Directions: The study highlights the need for further research into the practical applications, user-friendliness, and safety of LLMs, with a continued focus on patient-centered care. The findings support the development of AI as a powerful complement to human expertise, potentially alleviating time pressures in healthcare.
Summary updated