TL;DR
Researchers from Google Research conducted a large-scale randomized study demonstrating that a structured, LLM-powered "visit preparation agent" significantly improves patient readiness and confidence for medical appointments. By guiding users through a specific clinical workflow, the AI achieved higher clinician-rated utility and patient satisfaction than a standard chatbot, all while maintaining a high bar for medical safety.
What Problem Does This Paper Address?
The modern clinical encounter is under immense pressure. In the U.S., primary care visits often last only 15–20 minutes; in other countries, they can be as short as 10 minutes. Patients frequently arrive at these brief appointments feeling anxious, forgetting key symptoms, or unsure of what to ask, leading to suboptimal care.
While millions of adults are already turning to general-purpose AI chatbots for health advice, the clinical community remains skeptical: Does an open-ended conversation actually help a patient prepare? Or does it just provide a "hallucinated" sense of confidence? This paper investigates whether a Large Language Model (LLM) can be transformed into a reliable clinical tool by wrapping it in a structured, patient-centered workflow.
Methods
The team developed a visit preparation agent powered by Gemini 2.5 Flash. Unlike a standard chatbot that simply responds to prompts, this agent was designed with a structured 10-step workflow (see Figure 1) that includes:
- Context gathering: Identifying the visit type and current concerns.
- Expectation setting: Explaining what likely happens during the visit (e.g., tests, physical exams).
- Information provision: Explaining medical concepts in plain language.
- Agenda building: Helping the patient prioritize their top three questions.
- Summarization: Generating a "prep sheet" for the patient to take to their doctor.
The researchers conducted a prospective, three-arm randomized study with 802 U.S. adults. Participants were assigned to either a Control (minimal prompting), a Standard visit prep agent, or a Concise variant. The study utilized a "dual-perspective" evaluation: blinded board-certified clinicians reviewed the transcripts for safety and utility, while patients provided self-reports on their preparedness and satisfaction.
Datasets
The study did not rely on existing public benchmarks. Instead, it generated a novel dataset of 866 health-related conversations across a wide range of clinical domains, including cardiovascular, metabolic, and musculoskeletal conditions (see Figure 3 for participant demographics).
For the clinical evaluation, a random sample of 272 completed sessions was reviewed by independent physicians. Patient outcomes were captured via immediate post-interaction surveys (n=460) and a 14-day follow-up survey (n=95) to see if the preparation actually influenced their real-world doctor visits.
Key Insights and Ideas
- Structure Trumps Raw Capability: Even though all study arms used the same underlying model (Gemini 2.5 Flash), the structured agent significantly outperformed the control. This suggests that for high-stakes domains like healthcare, the interaction design and system prompting are just as critical as the model's parameters.
- The "Divergence" of Perspectives: One of the most striking findings was that clinician ratings of quality did not correlate with patient satisfaction. A patient might feel very helped by a conversation that a doctor finds clinically incomplete, and vice versa. This highlights the necessity of "dual-perspective" evaluations for medical AI.
- Utility without Safety Trade-offs: A common fear is that making an AI more "helpful" (proactive, detailed) increases the risk of "unsafe helpfulness" (giving bad medical advice). This study showed that the structured agent increased clinical utility (comprehensiveness and alignment) without compromising the high baseline of medical safety (over 90% in all arms).
Key Results
The structured visit preparation agent showed clear statistical superiority over the control:
- Clinician Perspective: The agent achieved a 16.6% higher rate of comprehensive preparation (67.2% vs. 50.6%) and was significantly better at aligning with the patient's specific health context (98.9% vs. 90.4%).
- Patient Perspective: Users reported 16.8% higher visit preparedness and 14.3% higher confidence in knowing what to ask their doctor. Overall satisfaction was also significantly higher (74.0% vs. 62.0%).
- Safety: Medical safety and appropriateness remained high (>93%) across all study arms, with no significant differences, suggesting the structured flow is a low-risk way to empower patients.
Limitations and Future Work
The authors acknowledge several hurdles for real-world deployment. First, there was a significant attrition rate, with roughly 42% of conversations ending before the full 10-step flow was completed. This suggests the need for even more streamlined or adaptive interfaces.
Second, while patients felt more prepared, the study had limited power to prove that this translated into better clinical outcomes (like improved diagnostic accuracy or treatment adherence) during the actual doctor visit. Future research should involve the clinicians who treat these prepared patients to see if the "prep sheets" actually improve the efficiency and quality of the face-to-face encounter. Finally, the study cohort was limited to Android users of the Google Health app, who may be more tech-literate than the general population, necessitating further studies on broader, more diverse demographic groups.
Summary updated