How Much Reader Disagreement in Ultrasound Is Made by the Probe?

When two doctors read the same ultrasound exam differently, we usually blame interpretation: the tissue is ambiguous, or the guidelines are vague. Ultrasound differs from CT or pathology in one important way. The image does not exist until someone presses a probe against the body at some angle. If the probe tilts away from the skin's surface normal (the direction perpendicular to the skin), sound scatters away from the sensor. Echoes are lost, boundaries blur, and shadows appear. Some of what looks like "inter-observer variability" may therefore be made upstream, at acquisition, before any reader sees the image. Retrospective archives cannot reveal this, because each patient in them was scanned once. This proposal runs a prospective repeated-acquisition study in thyroid ultrasound. Each patient is scanned several times under controlled conditions, including by a robot that keeps the probe perpendicular using live image-quality feedback. Several readers then read every scan. The goal is to measure, for the first time on patients, how disagreement splits into an acquisition part and an interpretation part, and whether enforcing perpendicularity removes the acquisition part specifically.

The two-sentence version is as follows. We scan the same thyroid repeatedly, by hand and by a perpendicularity-controlled robot, and have several readers read each scan, so we can separate "disagreement caused by how the scan was taken" from "disagreement caused by how it was read." This matters beyond robotics. It tells multi-reader models what their uncertainty actually represents, and it names a source of the retrospective-to-prospective performance gap that curated archives hide by construction.

Core Claim

Our hypothesis has two parts. First, a measurable share of between-reader disagreement on thyroid ultrasound comes from variation between acquisitions. Second, closed-loop perpendicularity control shrinks that share, beyond what robotic repeatability alone achieves, while leaving interpretation disagreement roughly unchanged. The file points to a specific motivating result. Can ultrasound confidence maps predict sonographers' labeling variability? showed that annotation variability tracks acoustic physics, as captured by confidence maps. These are per-pixel scores, computed by modelling sound propagation as a random walk, that run from 0 in shadow to 1 for clean transmission. That study worked on frozen volumes that had each been labelled many times, so it could not ask whether a better-aimed probe would have removed the variability. We ask exactly that question.

The quantity of interest comes from a crossed random-effects model, the standard tool for splitting measurement variance. Let $y_{par}$ be a reader's measurement, such as thyroid lobe volume, for patient $p$, acquisition $a$ of that patient, and reader $r$. We model it as

$$y_{par} = \mu + u_p + v_{a(p)} + w_r + (vw)_{a(p)r} + \epsilon_{par},$$

where $u_p$ is true patient variation, $v_{a(p)}$ is the acquisition-within-patient effect, $w_r$ is reader bias, and $(vw)$ is the reader-by-acquisition interaction. Each term has its own variance, $\sigma^2_u$, $\sigma^2_v$, $\sigma^2_w$, $\sigma^2_{vw}$. Our claim is that the acquisition fraction $\sigma^2_v/(\sigma^2_v+\sigma^2_w+\sigma^2_{vw}+\sigma^2_\epsilon)$ is substantial under freehand scanning. We further claim it falls under perpendicularity control, and that it falls more than under a robot following the same path with feedback switched off. A single-acquisition archive cannot estimate $\sigma^2_v$ at all, because it is absorbed into patient and residual variance. That is the concrete sense in which retrospective data hide part of the prospective gap.

Prior Work and Where This Sits

The robotic methods supply our instrument. Optimization of ultrasound image quality via visual servoing steers the probe by driving the angle of the confidence-weighted image centroid to zero while holding contact force constant. It converges to within 0.078° on a phantom and 0.050° ex vivo, but it controls only in-plane rotation. Precise Repositioning of Robotic Ultrasound uses confidence optimization to reposition the probe against motion. RSV: Robotic Sonography for Thyroid Volumetry is the closest work. It centres the thyroid and re-orients the probe around shadows on a phantom, reducing expert volume error from 20.85% to 8.23% and novice error by 85%. RSV measured error against a known phantom volume. It did not measure the spread of readings among human readers on real patients, and it did not separate a perpendicularity effect from a robotic-consistency effect. We reuse its task and measurement pipeline. We add repeated acquisitions, a panel of multiple readers, human patients, and an ablation with the feedback loop switched off.

The multi-reader modelling work constrains how we interpret results. The Latent Doctor Model learns a latent "annotator style" and predicts the distribution of labels a reader panel would give, reaching Spearman correlation up to 0.728 with disagreement among 20 pathologists. More is more: virtual expert panel trains one network per expert and averages them, lowering calibration error from 0.183 to 0.050. Both assume the image is fixed and that disagreement lives in readers or tissue. In ultrasound, that assumption silently bundles acquisition noise into what the models call "ambiguity." We test whether their predicted uncertainty tracks $\sigma^2_v$ or $\sigma^2_w$.

The validation literature motivates the prospective design. Mind the Performance Gap broke a deployed model's drop in AUROC (area under the ROC curve, a ranking-accuracy score) from 0.778 to 0.767 into infrastructure and temporal shift. Pipeline shift cost 0.016 AUROC, and the pipeline failed on 10 of 356 days. We add a third, ultrasound-specific component, acquisition shift, and we log the analogous infrastructure failures of a robotic system, such as dropped frames and control-loop delays. AI support for accurate and fast radiological diagnosis of COVID-19 showed that AI assistance raised Cohen's kappa among CT readers from 0.34 to 0.58. CT involves no acquisition operator, so that result cannot say whether ultrasound gains would come from better images or better reading. Our design can.

Research Program

We build a perpendicularity controller that is good enough to serve as an instrument; it is not meant as the contribution. A depth camera gives an initial skin-normal estimate, which prior optical profiling work reports holding within 2–3° (Development of an Operating Robot System with Sensor Feedback). The in-plane angle is then refined with the confidence-centroid law from Chatelain et al. For elevational tilt, the out-of-plane axis that Chatelain et al. leave uncontrolled, we compare mean confidence across small alternating tilts during the sweep and step toward the higher side. Contact force is held by impedance control at a level in the plateau Chatelain et al. report, where confidence stops rising above roughly 1 N. The full 6-degree-of-freedom manifold formulations such as MeshDMP are out of scope. We only need the probe to be measurably more perpendicular than by hand, and we verify this directly by logging estimated incidence angle per frame.

The study has three stages with the same structure. Stage 0 uses the CIRS Model 074 thyroid phantom, which has a CT reference volume of 30.01 ml. We first reproduce RSV's numbers, then add repeated acquisitions. Stage 1 recruits healthy volunteers, in the style of the open-source thyroid dataset used by Bozorgpour et al., where every acquisition condition can be repeated. Stage 2 is a prospective cohort in a thyroid clinic. Each patient receives repeated acquisitions under two of the four conditions, assigned by a randomized incomplete block design to keep the exam short. Stage-1 variance estimates set the Stage-2 sample size.

Every acquisition is read by a panel of sonographers and radiologists, blinded to condition. The primary readout is lobe volume. Secondary readouts are lobe and nodule segmentations, scored by Dice overlap between readers, and a categorical nodule margin/echogenicity rating, scored by Fleiss' kappa. Confidence maps and incidence angles are logged frame by frame. We can therefore also ask whether a scan's mean confidence predicts its acquisition deviation $v_{a(p)}$, which bears directly on the cheap alternative discussed below.

Baselines

The naive reference point is standard care: freehand scanning by a clinic sonographer, analysed with the same variance model. It is run on the same patients and read by the same panel, so every other condition is compared to what the clinic produces today. Any robotic condition must show a smaller acquisition fraction than this to matter at all.

The strongest existing robotic method is an RSV-style controller with shadow avoidance and lateral centring (RSV). On the phantom it reported 8.23% expert volume error and a 1.00 ml standard deviation across repeat scans. We reimplement it on the same arm, probe, and force setting, with the same segmentation network and sweep length. Our condition beats it if, on Stage 2 patients, the acquisition variance component is lower with non-overlapping bootstrap intervals, and it matches or improves on RSV's 1.00 ml repeatability on the phantom. We found no reported result on Lacuna for inter-reader or inter-acquisition variance of thyroid volume on human patients under any robotic system, so Stage 1 establishes that number.

The ablation that isolates our key ingredient is the same robot following the identical recorded path with confidence feedback disabled. This keeps robotic steadiness, force control, and sweep geometry, and removes only the perpendicularity correction. If this ablation matches the servoed condition, the benefit comes from repeatability, not perpendicularity.

The strongest software-only competitor is a freehand sonographer shown a live confidence-map overlay and asked to re-acquire when mean confidence drops below a threshold. The threshold is tuned on Stage 1 volunteers with the same tuning budget used for the controller's gains. On the analysis side, the strongest statistical competitor is a Latent Doctor Model (LDM) trained on the multi-reader freehand scans, whose reported Gleason KL divergence is 0.461 in a different domain. We evaluate it against the variance decomposition rather than against the robot.

Why not just…?

Why not just gate bad frames with a confidence display and let the sonographer re-scan? This is the most serious threat. If the frames most at risk are exactly the low-confidence ones, a display costs almost nothing and may capture most of the benefit. The physics argument in Chatelain et al., that scattered echoes are never received, shows that bad frames are unrecoverable afterwards. It does not show that a human prompted by a meter cannot avoid them. We therefore run this as a full arm, not a straw man. If its acquisition fraction matches the servoed robot's, we will report that the robot is unnecessary for this purpose. The per-acquisition regression of $v_{a(p)}$ on mean confidence tells us in advance how much gating could possibly help.

Why not just model the disagreement statistically with a latent-doctor or virtual-panel model? These models predict the spread of labels well (LDM; virtual expert panel). They cannot tell whether that spread would vanish with a better scan, because they only ever see one acquisition per patient. We test this directly by correlating per-case LDM entropy with the estimated acquisition and reader components. If entropy tracks the acquisition component, these models are partly reporting fixable scan quality as irreducible ambiguity. That would be useful for deployment, but it is not a substitute for fixing the scan.

Why not just use any robot, since steadiness alone might be what helps? RSV's own ablation found open-loop linear scanning worse than closed-loop correction. That comparison, however, was on a phantom and measured against ground truth, not reader spread. Our path-replay ablation settles the question on patients. We state in advance that equal performance between the replay and servoed conditions would count against the perpendicularity claim.

Proposed Experiments and Decisive Checks

The first check is whether the acquisition fraction is non-trivial under freehand scanning at all. If Stage 1 shows it is negligible, the question is answered and the program stops, with a reportable negative result: reader disagreement in thyroid volumetry is interpretive. The second check is the ordering of acquisition fractions across freehand, confidence-gated freehand, path replay, and servoed robot, with bootstrap intervals over patients. The third check tests whether reader and interaction variance stay flat across conditions. If they fall, part of the disagreement we attributed to interpretation was actually driven by image quality, which is itself informative. The fourth check is the LDM entropy correlation described above. The fifth concerns infrastructure. Following Mind the Performance Gap, we log control-loop delay, dropped frames, and aborted sweeps per session. We report how often the servoed condition degraded to effectively uncontrolled scanning, and whether sessions with delay spikes show larger incidence angles and larger $v_{a(p)}$. A robot that is perpendicular on the bench but frequently loses its loop in clinic would reproduce exactly the retrospective-prospective gap this proposal studies. We would count that as a finding, not a failure to hide.