Rater Disagreement over Gas Pockets, Not Gas Boundaries, in Free-Air CT

On abdominal CT, gas is the easiest thing to find. Air sits near −1000 on the Hounsfield scale, the standard CT density scale, while soft tissue sits near zero, so the edge of an air pocket is almost a step function. Free air outside the bowel signals a perforated gut that needs surgery, and the hard question is rarely where the gas is. The hard question is which gas it is. A given pocket may be free air in the peritoneal cavity, ordinary gas inside a bowel loop, retroperitoneal air, gas under the skin, or gas inside an abscess. Radiologists disagree on exactly this point. The best published detector, PACT-3D, is a 3D U-Net trained on a single consensus mask. Its authors cite discrepancy rates of up to 30% between junior residents and attending radiologists. They also report that the model's sensitivity rises from 0.81–0.83 to 0.92–0.98 once cases with less than 10 ml of free air are removed.

We propose to model inter-observer disagreement at the level of discrete gas pockets rather than at the level of pixels. The physics makes this cheap. A lenient density threshold plus grouping of connected voxels turns every scan into a few hundred candidate pockets. Raters then label each pocket by compartment instead of tracing outlines. A model predicts, for each pocket, the full distribution of rater labels, using image appearance and its distances to segmented organs. Our hypothesis is that much of the small-volume blind spot is a labeling artifact. Majority-vote masks erase the tiny, contested pockets, which teaches the network to suppress exactly the early-perforation signal that matters. Training on the rater label distribution should restore sensitivity at matched specificity, and it should also report when a positive call rests on contested evidence.

Core Claim

Existing multi-rater segmentation methods share one assumption: disagreement shows up as uncertainty about where a boundary lies. These include the variational model of Liu et al., the rater-specific latent Gaussians of Pionono, the mixture of stochastic decoders in MoSE, and the recurrent calibration of MrPrism. Their evaluations reflect that assumption, on blurry lung nodules, brain structures and the abdominal organs in QUBIQ. QUBIQ's thresholded overlap metric, for example, is dominated by large structures.

For free gas the boundary is nearly fixed by physics, and disagreement is categorical and object-level: whether a given pocket counts as free air at all. We claim three things. First, rater disagreement on gas concentrates on small pockets. Second, consensus training therefore suppresses small pockets. Third, a pocket-level model trained on rater label distributions recovers sensitivity on cases under 10 ml without losing specificity, in a way that pixel-level soft labels and ensembles do not. The first claim is directly testable before any model is trained. If it fails, the sensitivity argument fails with it.

Research Program

Annotation study. We will assemble a multi-rater abdominal CT cohort, because no public multi-rater pneumoperitoneum set appears in our sources. The cohort will be enriched for small-volume positives and for the mimics PACT-3D names as false-positive sources: cutaneous emphysema, abscess gas and retroperitoneal air. Candidate pockets come from a lenient density threshold, set below the strict air value to catch partial-volume bubbles, followed by grouping of connected voxels. A small learned proposal network adds any bubbles the threshold misses.

Each pocket is labeled by several readers drawn from both residents and attendings. The labels are free intraperitoneal, retroperitoneal, intraluminal, extraperitoneal mimic (subcutaneous or abscess), or artifact. One attending also traces full masks on a subset, so that we can audit candidate recall. Clicking one compartment label per pocket costs far less than slice-by-slice tracing, which the brain-tumor segmentation survey A survey of MRI-based brain tumor segmentation methods identifies as the core bottleneck of manual delineation. That saving is what makes five or more raters per scan affordable. We will measure annotation time against tracing.

Model. Each pocket is described by a 3D image crop around it, its volume and density statistics, and signed distance maps to nearby anatomy. A signed distance map gives, at every voxel, its distance to an organ surface, with the sign marking inside or outside. The anatomy comes from a whole-body organ segmenter in the style of MRSegmentator, covering bowel, abdominal wall and retroperitoneal organs.

The distance-map idea is borrowed from Jin et al.. They reduced esophageal target-volume delineation error by feeding distance maps of tumors, lymph nodes and organs at risk to the network, reaching 83.9% Dice. We apply the same idea to a different decision: whether gas is contained by a wall. One caution applies. MRSegmentator reports Dice of only about 0.64 on small vessels, so organ masks are imperfect. We will test sensitivity to mask quality.

For pocket $c$, let $\hat{p}_c$ be the predicted distribution over the $L$ compartment labels and $y_c$ the empirical distribution of the raters' labels. Training minimizes

$$\mathcal{L} = \sum_c w(v_c)\, \mathrm{CE}(y_c, \hat{p}_c),$$

where $\mathrm{CE}$ is cross-entropy and $w(v_c)$ is a weight that depends on pocket volume $v_c$. This weight keeps the many tiny pockets from being swamped by a few large ones.

As an optional second stage, we add a rater embedding in the spirit of Pionono, so the model can separate a resident's style from an attending's. We treat this stage as secondary: it should only stay if it improves agreement prediction on held-out raters.

Patient-level decision. A scan's free-air score aggregates its pockets' free-air probabilities. Because every pocket carries a predicted agreement level, a positive call can be reported as "supported by contested pockets only." That is a triage signal the consensus-trained detector cannot give.

Baselines

Naive reference. The do-nothing reference is the density threshold with grouping of connected voxels, calling every pocket outside the organ masks free air. It shows how far geometry alone goes. Its failure on intraluminal gas, which has the same density, sets the floor.

Strongest existing method. The strongest competitor is a PACT-3D-style 3D U-Net trained with the Dice plus focal loss described in PACT-3D on majority-vote masks. PACT-3D reports sensitivity 0.81–0.83, specificity 0.97–0.99 and segmentation Dice 0.81, with sensitivity of 0.92–0.98 once cases under 10 ml are excluded. Its weights and training data are not available to us, so we will reimplement it on our cohort with the same training scans and the same tuning budget. Our version will almost certainly be weaker than the original, which was trained on over 140,000 scans. We will therefore report the published numbers only as context and compare against our reimplementation. Beating it means higher sensitivity on cases under 10 ml at matched specificity of 0.97, with no loss on larger cases.

Strongest multi-rater competitor. The QUBIQ organizers report that ensembles of deterministic models often beat probabilistic models at matching rater disagreement. We will therefore train a deep ensemble of five 3D U-Nets on per-rater masks, which we can rasterize from pocket labels. We will also train MoSE-style stochastic decoders on the same masks. Neither has a reported result on gas: no reported result found on Lacuna for multi-rater methods on pneumoperitoneum.

Ablations. Two ablations isolate our key ingredient. The first keeps the pocket pipeline but trains on majority-vote pocket labels, which isolates the value of the rater distribution. The second trains a pixel-level U-Net on averaged soft masks, which isolates the value of working at the pocket level. A third ablation removes the distance maps. All methods receive equal GPU-hours for hyperparameter search.

Why not just…?

Why not just train a U-Net on averaged soft labels? This is the most serious threat, since it also preserves disagreement. Our hypothesis is that pixel-level losses weight a 0.2 ml bubble by its handful of voxels, so contested tiny pockets still contribute almost nothing to the loss. The pixel soft-label ablation tests this directly. If it matches our pocket model on sensitivity under 10 ml and on agreement calibration, the pocket decomposition adds only annotation savings, and we will say so.

Why not just ensemble 3D U-Nets, as QUBIQ suggests? Ensembles capture model variance, not the structured resident-versus-attending split. The QUBIQ summary itself flags the memory and compute cost of full-volume 3D ensembles. Still, an ensemble trained on per-rater masks is a fair competitor and is included at equal compute. If it wins, the conclusion is that pocket-level modeling is unnecessary.

Why not just lower PACT-3D's decision threshold? Lowering the threshold trades specificity for sensitivity along the same curve. Our claim is that the curve itself moves. For that reason every comparison is made at matched specificity, and we also report the full ROC curve for cases under 10 ml.

Why not just density thresholding? Intraluminal gas has the same density as free air. That is why the threshold serves only to propose candidates in our pipeline, and why it appears as the naive floor.

Proposed Experiments and Decisive Checks

First check: where disagreement sits. Before modeling, we test whether disagreement concentrates on small pockets. We measure per-pocket rater entropy against volume and against compartment, and resident–attending agreement (Cohen's kappa) by volume bin. If small pockets are labeled consistently, the label-erasure hypothesis is falsified. The study then reduces to a calibration contribution.

Second check: candidate recall. We compare candidate pockets against the attending-traced subset, with recall reported by volume bin. Any bubble the proposal stage misses is unrecoverable by the pocket model, so low recall at small volumes would cap everything downstream.

Main comparison. The main result is patient-level sensitivity at specificity 0.97, stratified at 10 ml and 1 ml, against all baselines. We use held-out raters and, if a second site is available, held-out scanners.

Calibration of disagreement. We test whether predicted label distributions match the held-out raters' distributions, using expected calibration error and Brier score per pocket. We also check the more clinical question of whether pockets the model flags as contested are the ones on which held-out residents and attendings actually split.

Secondary check on another dataset. As a check that pocket-level disagreement modeling transfers beyond gas, we will run the same comparison on LIDC-IDRI lesion-level labels. There, MoSE has been evaluated only on boundary distributions.