Lung-clearance treatments such as bronchoalveolar lavage (washing out mucus plugs and debris through a bronchoscope) and prone positioning (turning ventilated patients face-down so gravity drains the back of the lungs) are often judged by one number. That number is how fast consolidated or collapsed lung shrinks between two CT scans. The best-sized evidence in this file is a pediatric pneumonia cohort. It reports that lavage raised consolidation absorption from 1.60 to 2.13 cc/day, with p = 0.040 across 76 matched pairs (Lung consolidation absorption time in 238 pediatric cases of mycoplasma pneumoniae pneumonia). Every one of those volumes was traced by hand, slice by slice, and the authors name manual tracing error as a limitation. The effect is half a cubic centimeter per day, and its p-value sits just under the usual cutoff. If two radiologists tracing the same scan pair would disagree by a similar amount, the trial's conclusion depends partly on who did the tracing.
We propose to measure that disagreement directly and then build a segmentation model that reproduces it. The model does not output a single consolidation mask. It outputs a distribution over how a panel of radiologists would trace a baseline scan and its follow-up. Each simulated reader keeps the same personal tracing style at both time points. The result is, for each patient, a calibrated uncertainty on the volume change itself, which can be carried into the treatment-effect estimate. The question is narrow and testable. Is reader-to-reader variation in consolidation change large enough, and uneven enough across patients, that modelling it per patient changes what we conclude about a clearance therapy?
Core Claim
The quantity that clinical clearance studies care about is a difference between two segmentations, not a single segmentation. Reader disagreement has two parts. One is a reader's habitual style, such as consistently including the hazy rim around dense consolidation. The other is case-specific ambiguity, such as consolidation that abuts pleural fluid or a tiny residual patch at follow-up. Style partly cancels in a within-reader difference. Ambiguity does not cancel, and it varies from patient to patient.
A model that represents each reader as a latent style vector can produce correct paired uncertainty, provided the same style vector is held fixed across both scans of a patient. A deterministic segmenter hides this uncertainty. A single reliability coefficient averages it away.
We expect three things. First, rater-coupled paired sampling will give well-calibrated intervals on per-patient absorption speed. Second, the per-patient variance will be strongly uneven. Third, weighting the treatment analysis by this variance will measurably change the confidence interval on the lavage effect relative to both an unweighted deterministic analysis and a constant measurement-error correction. If the variance turns out to be roughly constant across patients, the probabilistic machinery is unnecessary, and we will say so.
Research Program
The work has three parts.
A multi-reader paired dataset. No such dataset is in the file. The probabilistic rater literature works on histopathology (Gleason 2019, ISUP), not lung CT. ATM'22 includes COVID-19 scans with consolidation but has single reference masks and no follow-up pairs (Multi-site, Multi-domain Airway Tree Modeling (ATM'22): A Public Benchmark for Pulmonary Airway Segmentation). We will therefore seek collaboration access to a serial-CT pneumonia cohort, ideally the pediatric mycoplasma cohort itself. From it we will draw roughly 60 baseline/follow-up pairs, chosen to cover lavage and non-lavage patients and the full range of consolidation size.
Five radiologists will independently trace consolidation and atelectasis on all 120 scans. Two of them will retrace a 20-pair subset after a washout period, which measures intra-reader inconsistency. Consolidation is far cheaper to trace than the up-to-7-hour airway trees described in Accurate Airway Tree Segmentation in CT Scans via Anatomy-aware Multi-class Segmentation and Topology-guided Iterative Learning, so this budget is realistic.
From these tracings we first answer descriptive questions that nobody has answered for this task. We measure how large between-reader variance in absorption speed is relative to the 0.53 cc/day effect. We measure how strongly each reader's bias correlates between baseline and follow-up. We identify which image situations drive disagreement, such as effusion adjacency, collapse, or small residual volume.
A rater-coupled longitudinal segmenter. We reuse the per-rater latent idea from Pionono (Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation). In Pionono, each rater is a Gaussian in an 8-dimensional latent space. The Gaussian's mean encodes systematic style and its spread encodes self-inconsistency. We also borrow the panel-simulation view of the Latent Doctor Model (The Latent Doctor Model for Modeling Inter-Observer Variability). Both were built and evaluated on single 2D histology images.
We change two things. The image backbone becomes a 3D CT segmenter that takes the registered baseline and follow-up together. More importantly, a sampled rater latent $z$ is drawn once per patient and decodes both time points. The predicted change for a simulated reader is then
$$\Delta V(z) = V_{\text{base}}(z) - V_{\text{follow}}(z),$$
where $V_t(z)$ is the consolidation volume produced at time $t$ under style $z$. Repeated sampling of $z$ gives a per-patient distribution of $\Delta V$, which we divide by the scan interval to get absorption speed. Sampling $z$ from the population prior simulates a new reader. Using a known rater's posterior reproduces that particular reader.
Uncertainty-propagating trial analysis. Each patient's sampled absorption speeds are treated as multiple imputations of an error-prone outcome. The matched treatment comparison is rerun over the imputations and the results are combined, which gives an effect estimate whose interval includes measurement uncertainty. A variant uses inverse-variance weights.
We keep the airway tree as a secondary, diagnostic layer rather than the core. ATM'22 shows that airway models lose branch detection on diseased scans, falling from about 90% to about 64%. This means automated lumen tracing cannot yet anchor a clearance endpoint in exactly the patients who need it. We will record whether disagreement concentrates in segments whose feeding bronchus is plugged. We will not build a joint airway–parenchyma model in this project.
Baselines
Naive manual reference. The naive reference is what the cited cohort did: a single reader's manual volumes, analyzed with an unweighted matched comparison (Lung consolidation absorption time in 238 pediatric cases of mycoplasma pneumoniae pneumonia). We reproduce it five times, once per reader. The spread of the five effect estimates and p-values is itself a result. Our method beats this reference if its interval covers the five-reader spread while a single reader's naive interval does not.
Deterministic segmenters. The strongest automated competitor is a deterministic 3D segmenter trained on the five-reader majority vote, with the same backbone, data split and tuning budget as our model. The file reports no consolidation-specific automated result. We therefore also run a fuzzy-connectedness tool in the style of CIDI-Lung-Seg (CIDI-Lung-Seg: A Single-Click Annotation Tool for Automatic Delineation of Lungs from CT Scans), and a fixed Hounsfield-unit threshold inside a lung mask. CIDI-Lung-Seg reports a mean overlap of 0.968 on LOLA11, but for whole lungs, not consolidation. No reported result was found on Lacuna for multi-reader consolidation volumetry or for paired volume-change uncertainty.
Strongest uncertainty-aware competitor. This is Pionono applied per scan, with independent rater sampling at each time point. Pionono reports an unweighted Cohen's kappa of 0.758 on Gleason 2019, against a prior best of 0.750. CLDM reports a KL divergence of 0.461 to the expert opinion distribution. Neither number transfers to CT. They establish only that these models are competitive in their original domain.
Ablations. The same model with independent $z$ per time point isolates our key ingredient, rater coupling. A version with no rater latent and Monte Carlo dropout uncertainty tests whether rater structure matters at all.
All learned models share the same folds, with rater-held-out and patient-held-out splits, the same GPU budget, and an equal number of hyperparameter trials. Beating these baselines means better coverage of held-out readers' absorption speeds at nominal levels, and higher rank correlation between predicted and observed per-patient variance.
Why not just…?
Why not just use a deterministic automated segmenter, which has zero reader variance? This is the most serious alternative. A fixed algorithm is perfectly repeatable, and repeatability may be all a trial needs. The catch is that repeatability is not agreement with what experts consider consolidation. A deterministic model can carry a case-dependent bias in exactly the ambiguous cases, such as effusion adjacency, where readers disagree. We will measure this directly as the deterministic model's absorption-speed error against the reader panel, split by patient. If that error is small and uniform relative to the 0.53 cc/day effect, the deterministic model wins and our contribution reduces to the reliability study.
Why not just run a small reliability substudy and apply a standard measurement-error correction with one intraclass correlation? This is also a real threat. Classical regression calibration needs only a single variance estimate. It is the right answer whenever measurement variance is roughly constant across patients. Our method only adds value if variance is uneven and predictable from the image. The decisive test is how much of the per-patient variance of panel disagreement our model's predicted variance explains on held-out patients. LDM's entropy reached a Spearman correlation of up to 0.728 with a 20-pathologist panel, which suggests such signal can exist. If the explained fraction is near zero, we will recommend the constant correction.
Why not just have three readers and use majority vote or STAPLE consensus? Consensus reduces noise but triples manual labor at every follow-up scan. It also collapses ambiguity into one mask, which the rater-modelling papers argue produces overconfident references at uncertain borders. In the airway setting, DeepTree shows that a consensus of tired readers can systematically omit true structure. The fair comparison is our model's predicted change against a three-reader consensus change on held-out patients. If consensus matches our interval coverage, the model's value is purely in saving labor, and the paper should be framed that way.
Proposed Experiments
Size of the problem. The first experiment comes from the five-reader data alone. We compute the between-reader standard deviation of absorption speed and the within-reader correlation of bias across the two time points. We then re-estimate the lavage effect once per reader. If the five p-values straddle 0.05, the motivation is established empirically, independent of any modelling.
Calibration. The second experiment uses leave-one-rater-out evaluation. The held-out reader's absorption speed should fall inside the model's predicted 80% and 95% intervals at those rates. The independent-sampling ablation should under-cover or over-cover, depending on the sign of the cross-time bias correlation we measured. If the coupled and independent versions calibrate equally well, the coupling claim fails.
Unevenness. The third experiment is the heteroscedasticity test described above, stratified by consolidation size and effusion presence.
Trial-level consequence. The fourth experiment applies the trained model to the remaining unannotated pairs of the cohort, if accessible. We compare the lavage effect interval under four analyses: single-reader manual, deterministic model, constant-variance correction, and rater-coupled imputation. We also run a power simulation that injects known effects into resampled patients and asks which analysis keeps nominal false-positive rates while detecting a 0.5 cc/day difference.
Transfer check. A secondary check applies the model to ATM'22 COVID-19 scans. There we can only score single-time-point agreement with the provided masks, but it tests whether the learned uncertainty rises in the opacity-heavy cases where airway models are already known to struggle.
References
- Lung consolidation absorption time in 238 pediatric cases of mycoplasma pneumoniae pneumonia
- Multi-site, Multi-domain Airway Tree Modeling (ATM'22): A Public Benchmark for Pulmonary Airway Segmentation
- Accurate Airway Tree Segmentation in CT Scans via Anatomy-Aware Multi-Class Segmentation and Topology-Guided Iterative Learning
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- CIDI-lung-seg: A single-click annotation tool for automatic delineation of lungs from CT scans