Did the Patient Understand That It Got Worse? Doubly Robust Measurement of Lay Comprehension of Disease Change

When a chest X-ray is compared with last month's, the most important thing a patient must take away is the direction of change: better, the same, or worse. Systems that explain radiology reports in plain language are judged on single, static reports, and their human evaluations draw on small convenience samples of readers who are more educated than the patients who most need help. This proposal builds an evaluation of longitudinal lay explanations whose core measurement is objective: after reading an explanation, can a reader correctly state the direction of change and what to watch for? It also supplies a statistical estimator that turns a small, biased set of human comprehension tests plus a large set of cheap simulated-reader judgments into an unbiased estimate, with valid confidence intervals, of how well a target patient population would understand.

The estimator is doubly robust, meaning it combines two models and stays correct if either one is right. The first model predicts a human's comprehension score from simulated "persona" readers, which are language models prompted to act as readers with a given literacy level. The second model predicts which reader–item pairs ended up with a human test. We argue that the bias in current lay-comprehension numbers is structured in a way that simple demographic reweighting cannot fix. Readers with low literacy are rare in crowd pools. Stable cases vastly outnumber worsening cases. Whether an item gets human-tested depends on its text. We claim these biases can reverse which explanation system looks best, and we design the study so that this claim can be checked against a deliberately representative reference cohort.

Core Claim and What Is New

The closest existing work, Doubly-Robust LLM-as-a-Judge, combines sparse human ratings with dense persona ratings. It uses an outcome model and a reweighting model estimated on text embeddings, with cross-fitting (fitting each model on held-out folds). It reports that the estimator "maintained a 95% confidence interval coverage even under heavy sampling bias" and stayed reliable with persona–human correlation as weak as ρ ≈ 0.23. It evaluates generic dialogue benchmarks (PSF, PRISM, DICES) with subjective ratings, on single turns.

We change three things. The outcome is an objective, answer-keyed comprehension test rather than a preference rating; it is keyed to disease direction from MS-CXR-T progression labels. The target population is defined by health literacy, which is exactly the dimension where convenience samples fail. The item distribution is longitudinal, where "stable" findings dominate and "worsening" cases are rare but decisive.

The motivating evidence comes from Two-Pronged Human Evaluation of ChatGPT Self-Correction. That study tested lay readers' severity judgments on 40 single report sentences, with an unadjusted participant pool. It found accuracy of 38.4% on raw text and 52.3% with chain-of-thought self-correction, and it documented that readers' confidence outruns their accuracy. Its lessons are that objective tests are necessary and that the measured numbers depend on who was sampled. We keep the objective test, move it to interval change, and correct the sampling.

Research Program

The benchmark. We start from MS-CXR-T, 1,326 image pairs with improving, stable, or worsening labels across five lung conditions, introduced with BioViL-T. We link each pair to its prior and current MIMIC-CXR reports. For each case, several systems generate a plain-language explanation of what changed. For each explanation, we write a short objective quiz: the direction of change, whether the finding is new or old, and one "what to watch" item. The answer key comes from the progression label, checked by a radiologist.

Clinical factuality of each explanation is scored separately, in two ways. RadGraph-based entity agreement, following RadGraph, checks the findings. Explicit temporal-claim agreement checks each stated direction against the label. The temporal check matters because a fluent explanation that says "stable" for a worsening case is the failure that DeltaNet and BioViL-T describe as normalcy bias and copying from prior reports.

The systems being evaluated. We compare four explanation systems:

  • a single-report simplifier in the style of the T5-large system from AEHRC at BioLaySumm 2025, which sees only the current report and so is blind to change unless the report itself states it;
  • a two-step pipeline that feeds the prior and current reports to that same simplifier;
  • a zero-shot large language model prompted with both reports and asked to write at a sixth-grade level;
  • the multi-agent self-correction pipeline from the Two-Pronged study, extended to receive both reports.

We are not proposing a new generator. The systems exist to create meaningful differences for the estimator to rank.

The estimator. Let a unit be a pair (reader profile $r$, explanation $j$) drawn from the target population. The target population combines a literacy distribution matched to published estimates that average US adult literacy sits between 6th and 8th grade, with a case mix matching clinical prevalence. Let $Y$ be quiz correctness and $S=1$ mark units that received a real human test. Let $X$ collect reader covariates (a short health-literacy screener score, education, age, primary language) and item covariates (progression label, condition, and an embedding of the explanation text). Let $\hat g(X)$ be an outcome model trained on human-tested units that uses the persona judges' answers as features, and let $\hat\pi(X)$ be the estimated probability that a unit received a human test. The estimand is mean comprehension $\theta$, and the estimator is

$$\hat\theta = \frac{1}{N}\sum_{u=1}^{N}\Big[\hat g(X_u) + \frac{S_u}{\hat\pi(X_u)}\big(Y_u - \hat g(X_u)\big)\Big],$$

summed over all $N$ target units. Its error scales with the product of the errors of $\hat g$ and $\hat\pi$, which is where the "either model can be wrong" guarantee comes from. We fit $\hat g$ and $\hat\pi$ with cross-fitting, as in double machine learning. We estimate system differences, $\theta_A-\theta_B$, directly and report intervals for them.

Two issues get explicit treatment. The first is overlap. Low-literacy readers will have tiny $\hat\pi$ in a crowd pool, and dividing by a tiny number makes the estimate unstable. We address this by targeted recruitment of that group, by reporting diagnostics for how well human-tested units cover the target units, and by trimming weights, with the trimming threshold fixed before the analysis. The second is the choice between the plain inverse-probability correction above and the Riesz-representer reweighting of the DR LLM-as-a-Judge paper. We run both. The Riesz version may be more stable when the text embedding is high-dimensional.

Baselines

Naive mean. The do-nothing reference is the unweighted mean quiz score from a convenience crowd cohort. This is how the Two-Pronged study and NoteAid (which used GPT-4 judges and medical students) evaluate. It uses the same human budget as every other estimator.

Post-stratification. The strongest cheap competitor reweights the crowd cohort to target shares of literacy tier by progression label. This is the method a careful survey statistician would use. It is fit on the same cohort and the same cells. If its bias and coverage match the doubly robust estimator, the extra machinery is unjustified.

Persona-only estimate. This averages the persona judges' answers over all target units, using no human data. It shows how far simulated readers alone get.

Prediction-powered inference. PPI++ and its reweighted variant RePPI use persona predictions to reduce variance and correct with human residuals, but without an item-level selection model. The DR LLM-as-a-Judge paper reports only that their coverage "dropped significantly" under sampling bias. We found no reported numeric result on Lacuna for PPI++ or RePPI on any clinical comprehension task. We run both on identical splits.

Ablations. Two ablations isolate the key ingredients. The first removes the explanation-text embedding from $\hat\pi$ and $\hat g$, keeping only demographic covariates. This tests whether text-dependent selection matters. The second removes the persona features from $\hat g$. This tests whether personas add accuracy or only reduce variance.

What counts as beating a baseline. The doubly robust estimator beats a baseline if, on held-out biased subsamples, it has lower absolute error against the reference-cohort value and near-nominal 95% interval coverage while the baseline undercovers. The key test is whether it recovers the reference ranking of the four systems in cases where the naive mean ranks them differently.

Clinical anchor. The strongest reported number for the underlying change-detection skill is CoCa-CXR's 65.0% average progression-classification accuracy on MS-CXR-T (CoCa-CXR). The file describes this as 4.8% above BioViL-T. It bounds how often any explanation can be expected to state the correct direction without access to the reports.

Why not just…?

Why not just post-stratify by literacy tier? This is the real threat. If selection into human testing depended only on literacy tier and case type, post-stratification would be close to optimal and simpler. Our design deliberately includes two mechanisms that post-stratification cannot capture. First, which items reach human testing depends on their text: long or jargon-heavy explanations get more dropouts. Second, persona judges can cut variance in thin cells, such as low literacy with worsening cases. The text-free ablation and the post-stratification baseline together settle the question. If neither loses to the full estimator on the reference cohort, we will report that post-stratification suffices, and that result is itself useful for clinical evaluation practice.

Why not just use simulated readers and skip humans? The DR LLM-as-a-Judge paper reports persona–human correlations as low as ρ ≈ 0.23. The Two-Pronged study shows that real readers' confidence diverges from their accuracy, a pattern that personas have no reason to reproduce. The persona-only baseline measures this gap directly against the reference cohort. We expect personas to overestimate low-literacy comprehension of worsening cases.

Why not just use readability formulas and clinical metrics? Paragraph-level Simplification of Medical Texts shows that human-written plain-language summaries still score at a Flesch-Kincaid grade of 13.11, which is evidence that such formulas track sentence length rather than understanding. BioLaySumm 2023 found readability and factuality pulling against each other. We report Flesch-Kincaid, Dale-Chall, and RadGraph agreement for every system and test whether they predict the estimated comprehension ranking. We expect them to miss explanations that are readable but get the direction wrong.

Why not just recruit a representative sample? That is the gold standard, and we do it once, as the reference. It is too expensive to repeat for every system iteration or hospital population, and low-literacy recruitment is slow. The purpose of the estimator is to reproduce reference-quality answers from a cheap biased sample plus personas. The reference cohort is what lets us check whether it does.

Proposed Experiments and Decisive Checks

We collect three datasets:

  • a reference cohort recruited with literacy quotas matched to the target distribution, which answers quizzes on a random subset of explanations from all four systems;
  • a convenience crowd cohort on the same items;
  • persona judgments at five literacy tiers for every target unit.

Following the protocol of the DR LLM-as-a-Judge paper, we then draw repeated biased subsamples from the pooled human data. Selection depends on literacy, on progression label, and on explanation length. For each subsample, we compute every estimator's bias against the reference value, its interval coverage and width, and its agreement with the reference ranking of systems.

We pre-register two predictions. The first is that the naive crowd mean overstates target comprehension most strongly on worsening cases. The second is that it ranks the current-report-only simplifier higher than the reference cohort does, because stable cases dominate and are easy. The proposal fails if post-stratification matches the doubly robust estimator on all criteria, or if the doubly robust estimator's coverage falls well below 95% once weights are trimmed as overlap requires. A secondary check tests whether temporal-claim factuality, rather than reading grade, best predicts estimated low-literacy comprehension across systems.