Calibrating Lung Ultrasound Raters Against Ventilator Physiology During Anesthesia

Under general anesthesia, most surgical patients develop atelectasis: patches of lung collapse caused by relaxed breathing muscles and abdominal contents pushing up on the diaphragm. Anesthesiologists respond by adjusting PEEP (positive end-expiratory pressure, a constant baseline pressure that holds airspaces open) and by performing recruitment maneuvers, which are brief high-pressure breaths that reopen collapsed lung. Their main bedside tool for judging whether these steps worked is lung ultrasound, scored by eye. Different clinicians score the same clips differently. On one atelectasis task, agreement between readers was only moderate before AI assistance (Schmidt et al. 2025). So the measurement that guides ventilation is partly a measurement of who happened to read it.

This proposal targets one feature of the intraoperative setting that ordinary multi-rater image datasets do not have: every scan is paired with ventilator physiology, and the ventilator settings are deliberately stepped within each patient. If PEEP is raised while insufflation pressure and table position stay fixed, the lung's compliance (volume gained per unit pressure) and the patient's oxygenation change mainly because lung tissue opens or closes. Those within-patient changes act as a physical reference. They can show which readers' outlines of collapsed lung track real change and which ones mostly add noise or compress the true effect. We would use this reference to re-weight readers, train a segmentation model on the recalibrated labels, and test whether the resulting atelectasis measurement predicts held-out physiology better than consensus labels do.

Core Claim

Within-patient interventional contrasts carry information about rater quality that spatial agreement cannot supply. These contrasts are pairs of scans of the same lung zone taken at different PEEP levels, with surgical conditions held constant. A simple measurement model makes the point. Let $y_{r,p,t,z}$ be the consolidated area that rater $r$ outlines in lung zone $z$ of patient $p$ at ventilation step $t$, and let $A_{p,t,z}$ be the unknown true area. We model

$$y_{r,p,t,z} = g_r A_{p,t,z} + b_r + \varepsilon, \quad \varepsilon \sim \mathcal{N}(0, s_r^2),$$

where $g_r$ is the rater's gain (does the rater register the full size of a change?), $b_r$ is a constant over- or under-calling bias, and $s_r$ is the rater's inconsistency. We tie the latent areas to physiology through

$$\Delta C_{p,t} = -\beta_p \sum_z \Delta A_{p,t,z} + \eta,$$

where $\Delta C_{p,t}$ is the change in respiratory compliance between consecutive steps, $\beta_p$ is a patient-specific positive scale, and $\eta$ is measurement noise.

Differencing across steps cancels $b_r$. Physiology therefore cannot recover the absolute boundary. It can recover the relative gains $g_r$ and the noise levels $s_r$, which spatial agreement alone cannot separate: two raters who share the same habit of under-calling would agree perfectly with each other and still both be wrong. This sets the scope of the claim. The method recalibrates the measurement of change, and change is the quantity that matters when titrating PEEP. Absolute bias is handled separately with a small anchor set, described below.

Relation to Prior Work

The closest existing idea is diagnosis-first ground truth, DiFF. DiFF weights annotators' masks so that a downstream disease classifier performs best, and it reports AUC gains of 2.04 to 2.73 percent across three tasks. It supplies our core intuition that label quality should be judged by usefulness for a downstream task. We differ in three ways. Our target is a continuous physical quantity rather than a diagnosis. Our supervision comes from deliberate within-patient interventions rather than between-patient labels. We also make the calibration an explicit, identifiable measurement model rather than end-to-end weighting. That last choice matters because end-to-end weighting against compliance could silently absorb non-lung effects such as abdominal pressure.

Two latent-rater methods supply machinery but motivate a different target. Zhang et al. 2020 learn pixel-wise confusion matrices for each annotator. Pionono gives each rater a Gaussian in latent space, whose mean captures bias and whose variance captures inconsistency, and reports a kappa of 0.758 on Gleason 2019. Both separate rater styles using image evidence and agreement only, so neither can say which style is physically correct. Our model adds an external signal that can.

Patel et al. 2025 collected per-pixel confidence from annotators on lung ultrasound. They found that training on masks thresholded at 60 percent confidence predicted oxygenation better than naive binarization. This motivates the project, because it shows that label handling changes physiological prediction. It also constrains it, because confidence thresholding is a cheap competitor we must beat. Patel's cohort was cardiac, not surgical, and did not model distinct raters.

On the clinical side, Liu et al. 2022 used manual 12-zone ultrasound scores in gynecologic laparoscopy. Atelectasis at 15 minutes in recovery was 40 percent after a recruitment maneuver versus 81 percent in controls, but the difference vanished by 24 hours. Wang et al. and Zhou et al. document how insufflation and head-down tilt stiffen the respiratory system. This is exactly why compliance changes must be taken at fixed insufflation and position: otherwise chest-wall loading would be mistaken for lung collapse.

Kamtam et al. 2025 replaced subjective CT collapse grades with nnU-Net lobar volumes after lobectomy. They found that intensity thresholds fail where collapsed lung meets the chest wall. Their tool supplies our absolute anchor on a CT subset, but their setting was static and postoperative and used a single consensus label.

Research Program

The work has three parts.

The first is data. In a prospective laparoscopic cohort, we follow a standard PEEP-titration protocol. Each patient is scanned in all 12 zones at baseline, after insufflation, at two or three PEEP steps under stable pneumoperitoneum, after a recruitment maneuver, and in recovery. Compliance, driving pressure, SpO2/FiO2, and arterial blood gases are recorded at each step. Six to eight readers of mixed expertise each segment consolidation in every clip and attach per-pixel confidence following Patel's protocol. Where postoperative chest CT is clinically indicated, Kamtam-style volumetry provides an absolute anchor for $b_r$.

The second is the calibration model. We fit the hierarchical model above by Bayesian inference, with compliance as the primary signal. SpO2/FiO2 enters either as a second, separately linked signal or is held out for evaluation, as described below. The posterior over true areas then becomes soft training targets for a segmentation network. The network uses the same backbone across all label schemes, so that only the labels differ.

The third is analysis. We report per-rater gain and noise estimates and check whether they are stable across random halves of patients. We test whether experience predicts gain, a result we expect: novices may under-register small reopenings. Finally, we ask whether Liu's 24-hour null result survives a recalibrated measurement, which bears directly on the recruitment paradox.

We deliberately do not pursue CT airway-tree modeling, the line of work behind ATM'22 and DeepTree. Intraoperative CT under stepped PEEP is not obtainable at scale, while ultrasound is already acquired at every step.

Baselines

The do-nothing reference is the current clinical practice of a single expert's manual 12-zone aeration score, as used by Liu et al. We will compare it against the same physiological endpoints. No correlation coefficient with compliance is reported on Lacuna for that score, so we will establish this number ourselves.

The consensus baselines are majority vote and STAPLE. Each produces training labels for the identical network, trained on the same patient splits with the same tuning budget, with hyperparameters selected on a shared validation fold.

The confidence baseline is Patel et al.'s 60-percent threshold, with confidence-weighted cross-entropy. Patel report it gave the lowest RMSE for SpO2/FiO2, but no absolute RMSE value appears on Lacuna. It is the strongest cheap competitor because it already improves physiological prediction without any rater model.

The strongest rater-modeling baseline is Pionono, trained on the same multi-rater masks, with its consensus or best-rater sample used as the label. Its reported figure, kappa 0.758 on Gleason 2019, is from a different domain and is not directly comparable.

The assisted-reading reference is Exo Lung AI's reported result on consolidation/atelectasis at a single lung window: Fleiss' kappa rising from 0.427 to 0.756 and sensitivity from 70.7 to 89.2 percent.

The key ablation fits our measurement model with the physiology term removed, so that rater parameters come from agreement alone. This isolates whether physiology contributes anything.

Beating the baselines means two things. Predicted per-step change in total consolidated area must explain held-out SpO2/FiO2 change, which was never used in calibration, with a confidence interval excluding the best baseline's value. The method must also be no worse than the baselines on agreement with CT anchor volumes.

Why not just…?

Why not just regress compliance directly from the ultrasound clips and skip segmentation? That model would predict the ventilator signal but give no map of where lung is collapsed. It could also latch onto shortcuts that vary with patients rather than with lung state, such as operator, probe depth, or body habitus. We will train this direct regressor as an additional comparison. If it predicts held-out oxygenation as well as our calibrated segmentation, we will report that honestly, and the remaining contribution reduces to spatial interpretability and rater diagnostics.

Why not just use DiFF with compliance as the downstream target? End-to-end weighting toward compliance cannot distinguish lung collapse from chest-wall stiffening caused by insufflation and tilt, which Wang et al. and Zhou et al. show is large. Restricting to within-condition contrasts and an explicit model makes clear what physiology can and cannot identify. We will run a DiFF-style variant trained on all scans, including across insufflation changes. We expect it to over-segment after insufflation, and the CT anchor subset will show whether that happens.

Why not just threshold confidence, as Patel did? This is a real threat, because confidence and accuracy may be correlated. Confidence is self-reported, however, and cannot reveal a reader who is confidently compressing every change. The ablation comparing physiology-calibrated labels with thresholded labels on held-out oxygenation decides this question. If the two tie, the extra machinery is not justified for training labels and survives only as a rater-audit tool.

Proposed Experiments

The first decisive check is synthetic. We simulate raters with known gains, biases, and noise on ultrasound-like masks paired with simulated compliance. We then verify that gains and noise are recovered, that bias is not recovered without anchors (confirming the identifiability argument), and how many patients and steps are needed. This tells us before collection whether a cohort the size of Liu's, about 40 patients, suffices.

The second check is held-out physiology. Calibration uses compliance only, and evaluation uses within-patient SpO2/FiO2 and arterial oxygen changes. Because the two signals have different confounders, agreement between them is not circular.

The third check is the absolute anchor. Postoperative CT volumes, quantified with Kamtam's pipeline, are compared with the ultrasound-derived recovery-room measurement under each labeling scheme, to confirm that the bias handling holds up.

The fourth check is reader impact. In a washout-crossover reading study modeled on Exo Lung AI, readers score with and without the calibrated model's maps. Success means reader kappa beyond the 0.756 reported with bounding-box assistance, together with tighter agreement in PEEP-step change estimates. The core idea fails if, on held-out oxygenation, the physiology term adds nothing over the agreement-only ablation and confidence thresholding.