Where Do Readers Disagree About Mild Fatty Liver? Separating Sampling, Signal, and Threshold Effects with Volumetric MRI References

Ultrasound is the default screening test for fatty liver, and it is known to miss mild disease, below roughly 30% fat. Sonographers looking at the same borderline image also often disagree. Both failures are usually blamed on the reader or on the limits of B-mode imaging, the grey-scale picture a standard scanner shows. This proposal tests a different, measurable explanation. Part of the "miss" and part of the disagreement may come from two things that have nothing to do with reader judgment. The first is spatial mismatch: the reference standard, liver fat measured by MRI, is usually averaged over the whole organ, while each ultrasound image samples only one fan-shaped slice of a liver whose fat can be patchy. The second is signal loss: depth attenuation, rib shadows, and speckle degrade some images more than others.

The plan is to register each standardized ultrasound view to the patient's full 3D MRI fat map. That gives a local fat value for the tissue the probe actually saw. We then fit a small statistical model of each reader's behavior that splits disagreement into three parts: each reader's personal threshold, noise caused by image quality, and error caused by comparing a local view against an organ-wide label. If the split holds up, it changes what a screening program should do. When the image is poor, the answer is to re-scan. When the image is good but the liver is near the diagnostic cut-off, the answer is to refer to a quantitative test. Neither case calls for more reader training.

Core Claim

We claim that in a cohort enriched for livers near the 5% fat threshold, a meaningful share of between-reader disagreement and apparent false negatives disappears once each image is scored against the fat fraction of the tissue it actually shows, and once image quality is estimated with a physics-based confidence map. We also claim that a classifier trained on these local labels will be more sensitive to mild steatosis than the same classifier trained on organ-averaged labels.

The pieces exist separately in prior work but have not been combined. Can ultrasound confidence maps predict sonographers' labeling variability? showed that acoustic confidence maps, computed with a random-walker model of sound propagation, track where a sonographer's repeated contours wobble. It studied anatomical segmentation of thyroid and muscle against manual contours, with no disease endpoint and no independent reference. The Latent Doctor Model and the virtual expert panel model individual raters, but only on 2D pathology slides, and they tie no rater behavior to a physical quantity. The liver steatosis systems either average 2D views against a biopsy or organ-level label (Accurate and Generalizable Quantitative Scoring of Liver Steatosis) or measure acoustic parameters in a small region of interest (CHESS2303; AI-QUS pilot). None of them asks whether the reference itself matches the sampled tissue. Our new ingredient is the volumetric registration of reference to view, combined with a rater model whose parameters have physical meaning.

Research Program

The data come from a prospective cohort built for this question, because the research file lists no public dataset that pairs multi-reader liver ultrasound with whole-liver MRI fat maps. We plan about 200 adults, enriched so that roughly half have organ-average MRI proton density fat fraction (MRI-PDFF, the MRI-measured percentage of fat in each voxel) between 2% and 12%.

Each participant has one ultrasound session and one MRI-PDFF scan on the same day. The ultrasound session follows the six standardized liver viewpoints and three images per view, the "rule of three" reported as sufficient for repeatable measurement by Accurate and Generalizable Quantitative Scoring of Liver Steatosis. Raw radiofrequency data are stored alongside the B-mode images where the scanner allows. A panel of about six sonographers, mixing junior and senior readers, grades every image independently and blind to MRI. Their grades are normal, mild, moderate, or severe, plus a free confidence rating. A subset of images is shown twice to measure each reader's agreement with themselves.

Registration is the step most likely to fail, so it is deliberately coarse. The liver is segmented on MRI with SegVol, which is reported to segment liver on MRI without retraining. The liver is then divided into its standard anatomical segments. Each ultrasound view is placed in this volume by its protocol viewpoint, then refined by matching a simulated ultrasound plane against the real image using the differentiable renderer of DiffUS. DiffUS turns a 3D volume into a simulated fan-beam ultrasound slice and can be optimized through by gradient descent. It was validated only under rigid alignment in brain surgery, so we use it only to choose a plane pose, not to recover voxel-exact correspondence.

The local fat value $f_i$ for image $i$ is the MRI-PDFF averaged over the insonated sector within the liver mask. We also compute its spread under pose perturbations, so that registration uncertainty is carried forward rather than hidden. The acoustic confidence map for each image, following the method of the confidence-map study, is summarized as a scalar $c_i$ over that same sector.

The analysis model is a per-reader psychometric curve, meaning the probability that a reader calls an image abnormal as a function of the true local fat. We add a lapse term that grows as image quality falls:

$$P(y_{ri}=1)=\tfrac{1}{2}\lambda(c_i)+\bigl(1-\lambda(c_i)\bigr)\,\sigma\!\left(\frac{f_i-\tau_r}{s_r}\right)$$

Here $y_{ri}$ is reader $r$'s binary call (steatotic or not) on image $i$. $\sigma$ is the logistic function. $\tau_r$ is the reader's personal fat threshold and $s_r$ its sharpness. $\lambda(c_i)\in[0,1]$ is a monotone decreasing function of confidence that represents guessing when the signal is unusable. An ordinal version handles the four grades.

We fit this model hierarchically and compare three versions: one where $f_i$ is the local value, one where it is the organ-average value, and one with $\lambda$ fixed to zero. Disagreement is then attributed to threshold spread (variance of $\tau_r$ across readers), signal-driven lapses, and label mismatch (the extra variance explained by local rather than global $f_i$).

In parallel we train an image model that predicts local fat and a full reader distribution, not a consensus label. Its backbone is the ordinal ResNet-18 design of the multi-view steatosis system. The confidence map is an extra input channel, as in the confidence-map study. The output head is conditioned on reader identity, in the spirit of the Latent Doctor Model.

Baselines

The naive reference point is what clinics do today: majority-vote B-mode grading scored against organ-average MRI-PDFF at the 5% cut-off. We report reader sensitivity and specificity for mild steatosis and Fleiss-style agreement. This row shows the size of the gap we are trying to explain.

The strongest learned competitor is the multi-view ordinal classifier of Accurate and Generalizable Quantitative Scoring of Liver Steatosis. It reports AUROC 0.85 for mild steatosis on its blinded histology cohort (HP-T). That figure is against biopsy on a different cohort, so it is context rather than a target. We re-implement the model and train it on our images with organ-average PDFF labels, using the same splits, the same augmentation, and the same hyperparameter search budget as our model. Beating it means higher mild-steatosis AUROC and sensitivity at matched specificity on held-out patients, scored against the same reference. The comparison is made both against organ-average PDFF and against local PDFF, so the gain cannot come merely from grading ourselves on our own label.

The strongest quantitative-hardware competitor is ultrasound-derived fat fraction (UDFF), a scanner-computed fat estimate from attenuation and backscatter. CHESS2303 reports AUC 0.90 for mild steatosis (MRI-PDFF ≥5%) in training, but AUC 0.66 for moderate steatosis in temporal validation. Where the scanner supports it, UDFF is acquired in the same session. We treat it as both a competitor and an object of the same analysis: we test whether its region-of-interest placement relative to local fat explains its errors.

For modeling reader distributions, the competitors are the Latent Doctor Model and the virtual expert panel. The latter reports expected calibration error (ECE, the average gap between predicted confidence and actual accuracy) falling from 0.183 for consensus training to 0.050, on esophageal pathology. Both are run on our images with identical backbones. Beating them means lower KL divergence (how far the predicted distribution is from the observed one) to held-out readers' grade distributions, and lower ECE.

The key ablations remove, one at a time, the local label (swapped for organ-average PDFF), the confidence channel and lapse term, and the DiffUS pose refinement (protocol viewpoint only). Each isolates one claimed ingredient.

Why not just…?

Why not just train on majority-vote consensus? The consensus label averages away exactly the signal we want to study, namely which disagreements are threshold-driven and which are signal-driven. The virtual expert panel work reports consensus training giving ECE 0.183 against 0.050 for a full panel. We run consensus training as a baseline, but it cannot answer the attribution question at all.

Why not just train a stronger 2D classifier on organ-average MRI-PDFF? This is the real threat. If spatial fat heterogeneity is small in practice, local labels add nothing, and the mild-steatosis gap is simply a capacity or data problem. The local-versus-global label ablation settles it directly. We also report the measured within-liver spread of PDFF across segments near the threshold. If that spread is small compared with registration error, we will say the sampling hypothesis failed and fall back to the reader-threshold and signal-loss decomposition alone.

Why not just use the Latent Doctor Model on ultrasound grades? It would capture reader styles, but its latent variables have no physical anchor. It cannot tell a lenient reader from a reader who happened to see a poorly penetrated image. The physics-free ablation of our model is essentially this baseline. The decomposition is worth something only if adding $f_i$ and $c_i$ reduces held-out KL and gives reader thresholds that stay stable across image quality.

Why not just replace readers with quantitative ultrasound hardware? UDFF and AI-based attenuation estimates already show high inter-observer ICC (0.96 in CHESS2303; 0.983 in the AI-QUS pilot, against 0.70 for Canon ATI and 0.30 for FibroTouch UAP). If the goal is only reproducibility, this is a strong alternative. But reproducible is not the same as correct. CHESS2303's temporal-validation drop to 0.66 AUC for moderate steatosis is what one would expect if the small measurement region sometimes sits on unrepresentative tissue. Our registration tests exactly that, so the project informs where hardware measurements should be taken rather than competing with them.

Proposed Experiments and Decisive Checks

The first check is controlled and runs before any clinical data. We build synthetic livers by assigning spatially varying fat to MRI-segmented liver volumes, map fat to acoustic properties, and render ultrasound views with DiffUS. We then inject known degradations: depth attenuation and simulated rib shadow. The reader panel grades these images. Because the true local fat and degradation are known by construction, we can test whether the fitted model recovers known heterogeneity effects and whether lapse estimates rise with injected degradation. If the model cannot recover planted structure here, the clinical decomposition is not trustworthy.

The second check is decisive on clinical data. On held-out patients, we ask whether local PDFF explains reader calls better than organ-average PDFF, measured as held-out log-likelihood of reader grades. We also ask whether the confidence term cuts the unexplained disagreement on low-confidence images while leaving high-confidence images unchanged. Self-agreement on repeated images provides a floor that no model should beat.

The third check is practical. We compare the local-label image model with the re-implemented multi-view baseline on mild-steatosis sensitivity at matched specificity. We also evaluate a triage rule that re-scans low-confidence images and refers high-confidence near-threshold cases. It is scored on how many false negatives it would have caught per extra MRI.

All results carry registration uncertainty through by resampling poses. We will report plainly if the claimed effect falls inside that uncertainty.