From Tool to Teammate: A Randomized Controlled Trial of Clinician-AI Collaborative Workflows for Diagnosis

Selin Everett, Bryan Bunning, Priyank Jain, Iván López, Anup Agarwal, Manisha Desai, Robert Gallo, Ethan Goh, Vinay B. Kadiyala, Zahir Kanjee, Jacob M. Koshy, Andrew Olson, Adam Rodman, Kevin A. Schulman, Eric Strong, Jonathan H. Chen, Eric HorvitzmedRxiv 2025 · DOI

Abstract

Early studies of large language models (LLMs) in clinical settings have largely treated artificial intelligence (AI) as a tool rather than an active collaborator. As LLMs now demonstrate expert-level diagnostic performance, the focus shifts from whether AI can offer valuable suggestions to how it can be effectively integrated into physicians' diagnostic workflows. We conducted a randomized controlled trial (n=70 clinicians) to evaluate the value of employing a custom GPT system designed to engage collaboratively with clinicians on diagnostic reasoning challenges. The collaborative design began with independent diagnostic assessments from both the clinician and the AI. These were then combined in an AI-generated synthesis that integrated the two perspectives, highlighting points of agreement and disagreement and offering commentary on each. We evaluated two workflow variants: one where the AI provided an initial opinion (AI-first), and another where it followed the clinician's assessment (AI-second). Clinicians using either collaborative workflow outperformed those using traditional tools, achieving average accuracies of 85% (AI-first) and 82% (AI-second), compared to 75% with traditional resources (p < 0.0004 and p < 0.00001). These results underscore the value of collaborative AI systems that complement clinician expertise and foster effective coordination between human and machine reasoning in diagnostic decision-making.

Introduction

While Large Language Models (LLMs) like GPT-4 have demonstrated expert-level performance on medical licensing exams, a persistent "interaction gap" remains. Previous research has shown a troubling trend: clinicians using these models often perform worse than the AI does on its own. This suggests that the primary hurdle is no longer the model's capability, but rather the failure of human-AI collaboration. Clinicians frequently struggle with effective prompting or fail to weigh AI suggestions appropriately against their own clinical judgment.

This study addresses this gap by shifting the paradigm from treating AI as a simple search tool to viewing it as a "teammate." By testing structured, collaborative workflows, the researchers aimed to determine if an AI acting as a peer reviewer could significantly enhance diagnostic accuracy and clinical reasoning.

Methods

The researchers conducted a randomized controlled trial (RCT) involving 70 U.S.-licensed physicians, including 39 residents and 31 attending physicians. The study utilized a custom GPT-4 system specifically designed to act as a "diagnostic collaborator."

Participants were randomized into three distinct diagnostic workflows:

  1. Traditional Resources: Clinicians utilized standard tools such as UpToDate, PubMed, or Google to solve cases.
  2. AI-as-First-Opinion (AI-first): Clinicians were presented with the AI’s initial assessment before forming their own diagnostic hypothesis.
  3. AI-as-Second-Opinion (AI-second): Clinicians worked through the case independently first, then shared their reasoning with the AI.
The experimental workflow randomized 70 clinicians into different diagnostic processes to compare traditional resources against AI-integrated workflows.

In the collaborative arms, the AI produced a "Synthesis of Reasoning." This output was presented as a table that highlighted points of agreement and disagreement, offering critical commentary on both the human and AI perspectives. This structured feedback was intended to foster deeper reflection and prevent simple over-reliance on the model.

The 'Synthesis of Reasoning' table integrates clinician and AI perspectives, highlighting areas of consensus and conflict to aid diagnostic decision-making.

The study utilized six complex clinical vignettes based on real-world patient cases. These cases were deidentified and had not been publicly released, ensuring the LLM could not rely on memorized training data. Responses were graded by board-certified physicians using a rigorous 19-point rubric developed from expert consensus.

Key Results

The results demonstrated a significant performance boost for clinicians using collaborative AI workflows.

  • Accuracy Gains: Clinicians in the AI-first group achieved 85% accuracy, while the AI-second group reached 82%. Both significantly outperformed the 75% accuracy achieved by the control group using traditional resources.
  • Actionable Decisions: The benefit was most pronounced in "clinically actionable" decisions, such as the final diagnosis and immediate next steps, where AI-assisted clinicians saw a nearly 15% increase in scores.
  • Efficiency: Interestingly, the AI-first workflow was the most efficient, averaging 631 seconds per case. The AI-second workflow took longer (723 seconds) because it required clinicians to perform independent work before the collaborative phase.
  • Attitude Shift: Exposure to the collaborative system improved clinician sentiment; 99% reported being open to using AI for complex reasoning post-study, compared to 91% beforehand.

Discussion and Insights

The study revealed several critical insights into the nature of human-AI collaboration in medicine:

Raising the Floor: The AI's primary impact was not necessarily making the top-performing doctors better, but rather reducing the "tail" of low-performing cases. It acted as a safety net, helping clinicians avoid significant diagnostic errors by suggesting alternative possibilities they might have otherwise overlooked.

The Sycophancy Trap: A notable observation in the AI-second workflow was "AI anchoring" or sycophancy. Despite instructions to be independent, the AI often mirrored the clinician's initial reasoning, even when that reasoning was incorrect. This highlights a technical challenge: models often prioritize being "helpful" or agreeable over being strictly accurate.

The "Better-Together" Paradox: While the collaborative workflows improved human performance, the AI alone still achieved a numerical accuracy of 90%—higher than any human-AI team. This suggests that humans are still "filtering out" correct AI suggestions, indicating that we have yet to find the optimal balance of trust and skepticism.

Conclusion

This trial demonstrates that moving from AI-as-a-tool to AI-as-a-teammate can significantly "raise the floor" of clinical performance. However, challenges like AI sycophancy and the "better-together" paradox remain. Future research should focus on "trust calibration"—training clinicians to identify when to trust the AI and when to override it—and developing system-level mechanisms to ensure AI models maintain their critical independence during collaboration.

Summary updated

Abstract

Early studies of large language models (LLMs) in clinical settings have largely treated artificial intelligence (AI) as a tool rather than an active collaborator. As LLMs now demonstrate expert-level diagnostic performance, the focus shifts from whether AI can offer valuable suggestions to how it can be effectively integrated into physicians' diagnostic workflows. We conducted a randomized controlled trial (n=70 clinicians) to evaluate the value of employing a custom GPT system designed to engage collaboratively with clinicians on diagnostic reasoning challenges. The collaborative design began with independent diagnostic assessments from both the clinician and the AI. These were then combined in an AI-generated synthesis that integrated the two perspectives, highlighting points of agreement and disagreement and offering commentary on each. We evaluated two workflow variants: one where the AI provided an initial opinion (AI-first), and another where it followed the clinician's assessment (AI-second). Clinicians using either collaborative workflow outperformed those using traditional tools, achieving average accuracies of 85% (AI-first) and 82% (AI-second), compared to 75% with traditional resources (p < 0.0004 and p < 0.00001; mean differences = 9.8% and 6.8%; 95% CI = 4.6%-15% and 4.0%-9.6%). Performance did not differ significantly between workflows or from the AI-alone score of 90%. These results underscore the value of collaborative AI systems that complement clinician expertise and foster effective coordination between human and machine reasoning in diagnostic decision-making.
BibTeX
@misc{everett2025from,
  author = {Selin Everett and Bryan Bunning and Priyank Jain and Iván López and Anup Agarwal and Manisha Desai and Robert Gallo and Ethan Goh and Vinay B. Kadiyala and Zahir Kanjee and Jacob M. Koshy and Andrew Olson and Adam Rodman and Kevin A. Schulman and Eric Strong and Jonathan H. Chen and Eric Horvitz},
  title = {{From Tool to Teammate: A Randomized Controlled Trial of Clinician-AI Collaborative Workflows for Diagnosis}},
  year = {2025},
  doi = {10.1101/2025.06.07.25329176},
  howpublished = {medRxiv},
}
Figure