Evaluating the Utility of Personal Health Records in Personalized Health AI

Rory Sayres, Kejia Chen, Ayush Jain, Matthew Thompson, Jonathan Richina, Xiang Yin, Jimmy Hu, Fan Zhang, Bob Lou, Mike Sanchez, Ines Mezerreg, Meredith Schreier, Hamsa Subramaniam, I-Ching Lee, Yugang Jia, Daniel Mcduff, Yossi Matias, Avinatan Hassidim, Dale Webster, Yun Liu, Jackie Barr, Quang DuongarXiv 2026 · arXiv

TL;DR

This paper from Google Research evaluates the utility of providing Personal Health Records (PHRs) as context to Large Language Models (LLMs) when answering consumer health queries. Using Gemini 3.0 Flash, the authors demonstrate that incorporating PHR data significantly improves the helpfulness and personalization of AI-generated health answers. However, they also uncover critical vulnerabilities—most notably "temporal blindness"—and show that an automated self-critique loop can dramatically mitigate these errors.

What Problem Does This Paper Address?

While consumer access to Personal Health Records (PHRs) has expanded, patient engagement remains low, largely because medical records are dense, fragmented, and difficult for laypeople to interpret. Concurrently, millions of adults are turning to AI chatbots for health advice. However, without access to a user's specific medical history, these models are forced to provide generic, boilerplate responses that often miss crucial clinical nuances or "red flags."

Conversely, simply dumping a complex, multi-year medical record into an LLM's context window introduces severe risks. Medical data is messy, chronological order matters immensely, and models can easily hallucinate lab values or misattribute family history to the patient. This paper systematically evaluates whether providing PHR context actually improves answer quality and maps out the specific error modes that arise when LLMs reason over complex longitudinal clinical data.

Methods

The researchers paired user health queries with de-identified real-world PHRs and evaluated the responses generated by Gemini 3.0 Flash under three experimental conditions:

  1. No PHR: The model answered the query in isolation (simulating standard chatbot use).
  2. Basic PHR: The model received a structured summary containing demographics, conditions, and current medications.
  3. Full PHR: The model received the complete, unstructured longitudinal clinical notes (averaging over 10,000 words).
Study Overview

To evaluate the outputs, the authors utilized two distinct frameworks:

  • SHARP Framework: An existing rubric assessing Safety, Helpfulness, Accuracy, Relevance, and Personalization.
  • PHR-Specific Rubric: A newly introduced 16-axis rubric designed to catch specific PHR interpretation errors across five categories (Fidelity, Robustness, Utility, Safety, and Equity).

Evaluation was scaled using an LLM-based "autorater" across the entire dataset, with a subset (n=95) validated by human clinician raters. Finally, the authors tested a remediation loop where a "Critic agent" used the automated rubric to flag errors and prompt the model to revise its own answers.

Datasets

The study leveraged two primary data components:

  • PHR Pool: 1,945 de-identified, real-world longitudinal records spanning 77,644 patient encounters across 9 US health systems.
  • User Queries (2,257 total): Drawn from three distinct distributions to mimic real patient behaviors:
  • Search Queries: Short, general health questions from the HealthSearchQA dataset.
  • Chatbot Templates: Synthesized, multi-turn style questions based on common Gemini app usage patterns.
  • Patient Calls: Real-world questions extracted and first-person rephrased from clinical telephone encounter logs between patients and nurses.

Key Insights and Ideas

  • Context Transforms Generic Search into Personalized Advice: Even for short, ambiguous web searches (e.g., "How do you get rid of phlegm?"), providing PHR context allowed the model to tailor advice based on underlying conditions like asthma or active prescriptions, shifting the output from an encyclopedia entry to actionable guidance.
  • The Chronology Trap (Temporal Blindness): The most prevalent error mode was the model's struggle with time. LLMs frequently treated discontinued medications as active, confused historical vitals with current ones, or suffered from "timeline logic failures"—such as treating a resolved abscess from months prior as the immediate cause for a recent lab culture.
  • Data Fabrication vs. Omission: In "Basic PHR" conditions, the model was more prone to data fabrication (hallucinating specific lab results to fill a void). In "Full PHR" conditions, it suffered more from nuanced clinical blind spots, such as missing a subtle mention of prior disease exposure buried deep in historical notes.
  • Self-Critique is Highly Effective for Clinical Alignment: A simple automated self-critique loop proved incredibly powerful. By forcing the model to evaluate its own draft against the PHR-specific rubric, the system corrected the vast majority of its contextual and temporal errors before outputting the final response.

Key Results

  • Massive Boost in Helpfulness: Both human clinicians and autoraters noted significant improvements when PHR data was included ($p < 0.001$). On a scale of [-1, 1], clinician ratings for overall helpfulness jumped from a baseline of 0.13 (No PHR) to 0.53 (Basic PHR) and 0.49 (Full PHR).
SHARP Ratings Framework Results
  • Safety and Accuracy Gains: While baseline safety ratings were already high for generic answers, the autorater identified significant improvements in safety constraints adherence and accuracy when grounded in PHR data.
  • Error Mitigation via Self-Critique: The self-critique loop achieved a 97.53% reduction in time awareness errors (dropping from 162 instances to just 4) and a 59.65% reduction in groundedness/confabulation errors.
PHR Ratings Framework Results

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope: The study focused strictly on queries that were clinically relevant to the accompanying PHR; it did not evaluate performance when users ask non-PHR-relevant questions within a PHR-aware session.
  • Data Modality: The context was limited to text summaries and unstructured notes, omitting other common PHR formats like PDFs, medical images, or structured FHIR data feeds.
  • Demographic Bias: The dataset was entirely US-based and skewed older, more female, and significantly whiter than the general US population.

Future work will need to explore how these models perform across international healthcare systems, evaluate the impact of AI-integrated PHRs on actual patient health outcomes, and refine long-context retrieval methods to prevent the model from anchoring on outdated historical events.

Summary updated

Abstract

Patient-managed Personal Health Records (PHRs) promises to empower patients to better understand their health; but information in the record is complex, potentially hindering insights. In this study, we assess the potential of large language models (LLMs, Gemini 3.0 Flash) to provide helpful answers to user health queries, when provided clinical data from PHRs as context. A total of 2,257 user queries were drawn from 3 different distributions to represent patient questions: shorter web search queries, longer questions derived from templates of chatbot conversations, and questions patients asked to their healthcare team (patient calls). Queries were matched with de-identified PHRs (from a pool of 1,945). Gemini responses were generated (1) without PHR context; (2) with a basic summary of demographics, conditions, and medications; (3) with full, extensive clinical notes. For evaluation, we leveraged an existing rating framework (SHARP), and developed a new framework for specific error modes when interpreting PHRs. Evaluation was performed using autoraters for the full set, and with clinician ratings for a subset (n=95), with both sets of raters knowing the full PHR context. We see significant improvements in the helpfulness of answers to all question types with PHR data (p < 0.001, paired t-test). We also observe potential gains in safety, accuracy, relevance and personalization of answers. Our PHR evaluation framework further identifies gaps in LLM understanding of particular aspects of complex PHRs, such as temporal disorientation, and rare but meaningful confabulations. These results suggest potential for PHR data to help people with a wide range of user needs; and provide a framework for monitoring for gaps in LLM answers based on PHR context. This study motivates further work to assess and realize potential benefits to users from understanding their health records.
BibTeX
@misc{sayres2026evaluating,
  author = {Rory Sayres and Kejia Chen and Ayush Jain and Matthew Thompson and Jonathan Richina and Xiang Yin and Jimmy Hu and Fan Zhang and Bob Lou and Mike Sanchez and Ines Mezerreg and Meredith Schreier and Hamsa Subramaniam and I-Ching Lee and Yugang Jia and Daniel Mcduff and Yossi Matias and Avinatan Hassidim and Dale Webster and Yun Liu and Jackie Barr and Quang Duong},
  title = {{Evaluating the Utility of Personal Health Records in Personalized Health AI}},
  year = {2026},
  eprint = {2605.18937},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.18937},
}
Figure