Which Disagreement Is the Device's Fault? Separating Hardware Noise from Clinical Ambiguity with Handheld Video

When a cheap handheld ultrasound probe or a smartphone ophthalmoscope produces an image that experts disagree about, the disagreement has three possible causes. The case may be genuinely ambiguous, so experts would disagree even on a hospital scanner. The device's per-frame noise may be hiding the feature, which recording more video would fix. Or the device's resolution may be too coarse to show the feature at all, which only a better machine would fix. These causes call for three different actions: accept the uncertainty, rescan longer, or refer the patient.

We propose a model that predicts how a panel of experts would split on a case captured with low-cost hardware, and that divides the predicted disagreement into these three parts. The key to the division comes from satellite data fusion. Noise that differs from frame to frame averages out as independent frames accumulate. Detail smaller than the sensor's footprint does not come back no matter how many frames arrive. So by measuring how expert disagreement falls as a handheld clip grows longer, and anchoring it to readings of the same patient on a hospital device, we can tell the three causes apart. We can then use the split to decide who needs referral.

Core Claim

Consider rater $r$ reading case $i$ on device $d$ from a clip with $K_{\text{eff}}$ effectively independent frames. We model the rater's perceived value of a diagnostic feature as

$$ s_{ird} = f_i + \epsilon_r + u_{rd}(i) + \frac{\eta_d(i)}{\sqrt{K_{\text{eff}}}}, \qquad y_{ird} = \mathbb{1}[s_{ird} > t_r]. $$

The symbols are as follows:

  • $f_i$ is the true feature value.
  • $t_r$ is the rater's personal threshold, which carries between-rater bias.
  • $\epsilon_r$ is the rater's own inconsistency, with variance $\sigma_r^2$.
  • $u_{rd}(i)$ is a rater-specific misreading caused by the device's resolution limit. It has variance $\tau_d^2(i)$ and is the same in every frame.
  • $\eta_d(i)$ is per-frame device noise, with variance $\sigma_d^2(i)$.
  • $K_{\text{eff}}$ is the number of effectively independent frames. We estimate it from frame-to-frame decorrelation, because a still probe yields many nearly identical frames.

The $1/\sqrt{K_{\text{eff}}}$ scaling is simply the standard error of an average of independent noise. By definition, the hospital device is the reference with $\tau \approx \sigma \approx 0$. Its panel disagreement therefore measures intrinsic ambiguity, and the handheld device's excess disagreement splits into a part that shrinks with $K_{\text{eff}}$ (noise) and a part that does not (resolution).

The claim has two parts. First, a network can predict $\tau_d^2(i)$ and $\sigma_d^2(i)$ for each case from the handheld clip alone. Second, referring patients by predicted resolution-limited disagreement recovers more of the disagreement that a hospital scan would resolve than referring by total predicted uncertainty.

What Exists and What Is Missing

Rater modeling, which we reuse. Pionono gives each annotator a Gaussian latent code, whose mean captures bias and whose variance captures inconsistency. The Latent Doctor Model encodes each doctor's style from a small set of cases that doctor has labeled. Virtual expert panels train one model per pathologist and ensemble them. All three were built on standardized histopathology from high-end scanners. None can say whether the spread they model comes from the patient or from the sensor, because the sensor never varies in their data. We reuse their per-rater latent machinery for $t_r$ and $\epsilon_r$ and add device terms they lack.

Evidence that motivates the problem. FUSEP shows that fetal ultrasound performance shifts with scanner vendor. It uses consensus boxes on frames extracted from video, so it discards both the rater spread and the temporal redundancy we need. The virtual-panel paper shows that collapsing raters to a consensus label is harmful: expected calibration error rises to 0.183, against 0.050 for the full panel. This constrains us to predict distributions over raters rather than consensus labels.

Temporal fusion, which we borrow. From remote sensing we take the idea of a sparse high-resolution reference plus a frequent coarse stream, as in MARSM's variational recurrent fusion. BAM-MRCD supplies the negative result that anchors our resolution term: fire scars much narrower than the coarse pixel stay poorly delineated (small-fire IoU of 27.25%) regardless of fusion. That paper also argues against upscaling before classification, because upscaling invents detail. Arclight mosaicking fuses shaky low-cost video into a wider field, but it was never checked against a gold standard or read by multiple experts.

Enhancement, which is the main competing philosophy. SSC U-Net and PET diffusion restoration improve cheap-device images by pixel metrics. The synthetic-CT TB study found that feeding the full synthetic volume into a classifier did worse than feeding two central slices, because of reconstruction artifacts. That result is the reason we treat enhancement as a competitor to test, not a component to trust.

Research Program

Paired multi-rater data. The central asset is a paired, multi-rater, multi-device dataset. No file resource contains one, so we collect it. The primary site is a thyroid-nodule clinic, chosen because thyroid ultrasound is already a multi-rater benchmark (TNMIX in DiFF) and has a precedent for handheld/cart pairing (the SSC U-Net cohort of 72 in vivo pairs). For about 250 patients, one visit produces three recordings: a cart-system scan, a handheld-probe cine clip of 10 to 20 seconds, and a second handheld clip from a different probe brand. Six sonographers read each nodule's scored features, including echogenicity, margin, and echogenic foci. Each nodule is read under three conditions: the cart image, a single handheld frame, and handheld clips truncated to several lengths. A Latin-square schedule ensures no reader sees the same patient twice within a washout period.

Controlled pilot. Before collecting patients, we run a pilot on existing multi-rater images. We inject synthetic speckle, which is per-frame and resampled independently, and synthetic blur, which is fixed across frames, at known levels into TNMIX and REFUGE-2 cases. Because the true split between noise and resolution is known by construction, this pilot tests whether readers' disagreement actually follows the $1/\sqrt{K}$ pattern for noise and a flat curve for blur. If readers do not behave this way, the model's premise fails early and cheaply.

Model. A per-frame encoder feeds an attention pooling module that also receives an estimate of $K_{\text{eff}}$. The module outputs a feature distribution $f_i$ and two case-level variance heads, $\hat\sigma_d^2(i)$ and $\hat\tau_d^2(i)$. Rater-specific thresholds and inconsistency come from a Pionono-style latent code per rater. The probability that rater $r$ calls the case positive integrates over the terms of the equation above.

Training uses the paired design. Cart readings fit $f_i$ and the rater terms. Handheld readings at several clip lengths fit the two device heads. The requirement that only $\hat\sigma^2$ may shrink with $K_{\text{eff}}$ is what makes the two heads identifiable. At deployment, only the handheld clip is available.

Cross-dataset validation. Validation targets transfer across hardware, not a random split. We train on one handheld/cart pair and test on the second probe brand. On FUSEP, we relabel 300 nuchal-translucency frames per vendor with five sonographers and test in leave-one-vendor-out fashion. The goal is to see whether the predicted resolution term flags nasal bone, which occupies only 0.2 to 0.5% of the image, as resolution-limited on every vendor.

Baselines

No-model reference. The naive reference point is a referral rule that uses no model. It refers the fraction of patients whose single handheld frame shows the most reader disagreement, or refers everyone below a fixed image-quality score. It fixes the floor for referral utility at no cost.

Strongest direct competitor. The strongest competitor is a multi-rater model with a device indicator added as input: Pionono, with the virtual-panel ensemble (Grove et al.) as a second variant. These are the best published methods for predicting how experts split, and giving them the device label lets them absorb device effects into total spread. Pionono reports unweighted Cohen's kappa of 0.758 on Gleason 2019. The virtual panel reports ECE of 0.050 and F1 of 78.2% on the Dutch Barrett's esophagus panel. Both numbers come from histopathology. For predicting rater distributions on paired handheld/cart ultrasound, we found no reported result on Lacuna. Both baselines get the same encoder, the same frames (pooled by mean), the same readings, and the same hyperparameter search budget as our model. Beating them means matching their KL divergence to the held-out panel distribution while producing a better referral curve, as defined in the decisive checks.

Enhance-then-read pipeline. Here SSC U-Net, trained on our handheld/cart pairs, cleans each frame, and a Pionono model trained on cart images reads the result. Its reported gain is a 3.23% point-target sharpness improvement over a plain U-Net, which is a signal-quality measure with no diagnostic readout.

Standard uncertainty split. A deep ensemble with test-time augmentation over frames splits uncertainty into the usual data-driven (aleatoric) and model-driven (epistemic) parts. This checks whether the textbook decomposition already recovers the noise/resolution distinction.

Ablations of the key ingredient. Two ablations target it directly. The first removes the $K_{\text{eff}}$ scaling constraint, so both variance heads are free. The second removes the paired cart anchor, so intrinsic ambiguity must be inferred from handheld data alone. If either ablation matches the full model, the frame-scaling idea is not doing the work.

For detection transfer on FUSEP, the reference is Relation-DETR at 85.6% mAP on crown-rump-length views, as reported by FUSEP. We report whether our encoder keeps detection within a few points of this while adding the attribution heads.

Why not just…?

Why not just predict total disagreement with a multi-rater model and refer the most uncertain cases? This is the real threat. If total predicted spread ranks patients for referral as well as our resolution term does, the decomposition is academic. Our argument is that total spread mixes three kinds of uncertainty. Intrinsically ambiguous cases gain nothing from a cart scan, and noisy cases need only a longer clip, so referring them wastes the scarce hospital slot. The decisive check measures this directly. If total-spread referral matches ours at a 20% referral budget, we will report that the split does not pay off in this setting.

Why not just enhance the handheld images and treat them as cart images? Enhancement can remove per-frame noise, but it cannot recover detail below the device's resolution without inventing it. The synthetic-CT TB study found that generated content hurt classification, and BAM-MRCD argues against upscaling before inference for the same reason. Enhancement also hides from the reader the fact that the detail was never measured. Our experiment reports two things for this baseline: its referral utility, and whether readers' disagreement on enhanced frames falls on resolution-limited cases. A drop on those cases would mean either that enhancement genuinely helps or that readers are being misled. We tell these apart by checking the readings against the cart readings.

Why not just use the standard aleatoric/epistemic split from an ensemble? Epistemic uncertainty reflects what the model has not learned. Aleatoric uncertainty lumps patient ambiguity and sensor noise together. Neither is tied to the physical behavior of repeated frames, which is what separates rescan from refer. In the controlled pilot, where the truth is known, we test whether ensemble aleatoric uncertainty tracks injected speckle, injected blur, or both.

Why not just train on consensus labels from the cart device and distill into a handheld model? A consensus target throws away the spread we are trying to attribute, and the virtual-panel results show that consensus training miscalibrates (ECE 0.183 against 0.050). We include this as a sanity baseline to confirm the same pattern appears in ultrasound.

Decisive Experiments

Scaling test in the pilot. In the synthetic-degradation pilot, the model's predicted $\hat\sigma^2$ should correlate with injected speckle and $\hat\tau^2$ with injected blur, with little cross-talk. Readers' measured disagreement should fall with clip length only under speckle. If readers' disagreement does not fall with more frames under speckle, the premise is wrong and we stop.

Attribution test on paired patients. On held-out patients, the measured drop in panel disagreement from handheld to cart should be predicted by $\hat\tau^2$ better than by total predicted spread. The measured drop from a one-frame to a full-length handheld clip should be predicted by $\hat\sigma^2$. Both are assessed by rank correlation. The matched KL divergence to the held-out panel and ECE confirm that the attribution heads do not cost predictive accuracy.

Referral test. At referral budgets from 10% to 30%, we compute the share of all handheld-attributable disagreement that referred patients' cart scans actually resolve. Our $\hat\tau^2$ ranking must beat total-spread ranking from the strongest multi-rater baseline, the enhance-then-read pipeline, and the ensemble split, across two held-out handheld probe brands. On FUSEP, leave-one-vendor-out, the resolution term must flag the same small structures on the unseen vendor. If the gain disappears on the unseen probe, the device heads have memorized one sensor rather than learned how resolution limits behave, and that is itself the result we would publish.