Which Contouring Disagreements Can Spectral CT Resolve?

Experts drawing the same organ on the same CT scan disagree, and they disagree most where the organ fades into its neighbors. The esophagus against mediastinal fat and the pancreas against bowel are typical cases. Current segmentation models treat this disagreement as irreducible noise: they learn to reproduce the spread of human contours, but they cannot say whether the spread comes from the image lacking the contrast needed to see the edge, or from experts following different conventions about where a margin belongs. This proposal uses dual-layer spectral CT to separate the two. In this scanner design, two stacked detector layers record low- and high-energy X-rays from a single exposure, so a conventional-looking image and energy-resolved images (tissue-composition maps such as iodine concentration) come from the same photons and are perfectly aligned.

Because of that alignment, the same experts can contour the same patient twice: once under conventional grayscale viewing and once under spectral viewing. Any change in their disagreement is then attributable to the added physical contrast, with no registration or motion confound. We would collect such paired-view, multi-rater labels. We would then train a memory-efficient 3D model that predicts, for new patients with no expert reads, two maps. The first shows where disagreement would collapse if readers had spectral contrast. The second shows where disagreement persists because it reflects convention. The first map tells an automatic contouring system where it should be sharp. The second tells it where it should honestly report a band.

Core Claim

Inter-observer variability along low-contrast boundaries has two components that current methods conflate. The first is an information-limited part that physical tissue-composition contrast can remove. The second is a convention-limited part that no image can remove. The claim is that dual-layer spectral CT, combined with a paired-view reader protocol, lets us measure this split directly. A single-pass 3D model conditioned on rater and viewing condition can then learn to predict the split from the spectral data alone.

Let $p_\theta(y \mid x, r, v)$ be the model's probability that a voxel belongs to the structure. Here $x$ is the full spectral input, $r$ indexes the rater, and $v \in \{\text{conv}, \text{spec}\}$ is the viewing condition the rater used. The predicted disagreement at voxel $i$ under condition $v$ is the variance across raters, $D_v(i) = \mathrm{Var}_r[p_\theta(y_i \mid x, r, v)]$. The resolvable component is then

$$ R(i) = D_{\text{conv}}(i) - D_{\text{spec}}(i), $$

and the residual $D_{\text{spec}}(i)$ is the convention component. The model always receives the spectral input, because the conventional image is a fixed function of the same data. The viewing condition therefore changes only which human behavior is being imitated, not what the model sees.

This is not the obvious "memory-efficient 3D spectral probabilistic segmenter" that the four directions suggest when stacked together. That system would be trained on labels drawn under one viewing condition. It could only mimic those raters' spread, and it could never tell which part of the spread the spectral data could have removed.

Relation to Prior Work

The motivation comes from two places. Spectral image clustering on dual-energy CT scans using functional regression mixtures clusters per-voxel spectral attenuation curves in 91 head-and-neck scans, reaching a median Dice of 0.66 against 0.15 for a Gaussian mixture. It argues qualitatively that a peri-tumoral zone visible spectrally is missed by manual contours. That suggests some disagreement is information-limited, but the paper models no raters and measures no disagreement. The multi-rater literature models disagreement purely statistically. Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures places annotator variation in a 6-dimensional latent space on 2D MRI slices. Pionono (Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation) gives each rater a latent distribution whose mean captures systematic bias and whose variance captures self-inconsistency, on 2D histology. We reuse Pionono's rater-conditioned latent as the method ingredient and add a viewing-condition variable. Neither paper can attribute disagreement to image physics, because each rater saw only one rendering of the data.

The physics constrains the claim as well as motivating it. Dual-layer detectors give perfect spatial and temporal co-registration of the energy channels (Phantom-based quantification of the spectral accuracy of lower extremity vascular imaging using a second-generation dual-layer spectral CT at 100 kVp and 120kVp). They support material decomposition into iodine and water bases and virtual monochromatic images, meaning images simulated at a single photon energy (Application of virtual monochromatic images in bone mineral density measurements: a phantom study). The co-registration property is what makes the paired-view protocol clean.

Against this, Feasibility study for proton dose calculation of esophageal squamous cell carcinoma based on stopping power ratio directly derived from dual energy CT found no significant difference in gross target coverage or mean organ dose between spectral and conventional planning in esophageal cancer. It did find a significantly lower gradient index, meaning a steeper dose fall-off at the target edge. We take this result seriously. It predicts that the resolvable fraction $R$ will be spatially concentrated at specific boundary segments rather than spread over the organ. Our analysis is designed to test exactly that, not to assume spectral data helps everywhere.

On segmentation of the target structures themselves, Segthor: Segmentation of Thoracic Organs at Risk in CT Images shows the esophagus is the hardest thoracic organ, with a Dice of 0.80. 3D Enhanced Multi-scale Network for Thoracic Organs Segmentation raises esophageal Dice from 0.7664 to 0.8597 with 3D auto-context. Both methods are single-energy and deterministic. The only dual-energy segmentation work in the file, Towards Automatic Abdominal Multi-Organ Segmentation in Dual Energy CT using Cascaded 3D Fully Convolutional Network, collapses the two energies into one linearly mixed image. It finds the best mixing weight differs by organ, which is evidence against fixed collapsing and for feeding material maps as separate channels.

Research Program

The first component is data. We would assemble a retrospective cohort of roughly 60 contrast-enhanced dual-layer spectral CT scans, split between thoracic scans (esophagus) and abdominal scans (pancreas), from a scanner class like the Philips spectral system used in the esophageal study. Four to five radiation-oncology or radiology readers would contour each structure twice. The first read uses a conventional-equivalent image. The second, after a washout period of at least four weeks and in randomized order, uses a spectral view: a low-energy monochromatic image plus an iodine map. To separate spectral benefit from simple re-reading, a subset of scans would be read twice under the conventional view. That gives an intra-rater test–retest control, so a drop in disagreement under the spectral view must exceed the drop from rereading alone.

The second component is measurement, which needs no learning. For each boundary point we compute the spread of rater surfaces in millimeters under each view. We then locate where the spread shrinks and test whether the shrinkage coincides with high spectral separability across the boundary, measured as the iodine/water contrast-to-noise ratio between the two sides.

The third component is the model. A 3D encoder–decoder takes the material-basis maps and two monochromatic images as input channels at the first layer. Early fusion of a handful of channels adds almost no memory, and we say so plainly. The memory problem in this project is not the spectral input. It lies in whole-volume context for thin tubular organs and in representing a distribution over raters. To fit large crops (on the order of 160×160×160 voxels) on a single 24 GB GPU, the encoder uses reversible blocks. These recompute activations during the backward pass instead of storing them, following Deeper Image Quality Transfer: Training Low-Memory Neural Networks for 3D Images, which kept deep 3D networks near 541 MB at the cost of a roughly 15× slower epoch. Where a full 3D convolution is too costly, the encoder uses depth-shifted thin kernels as in E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation. Rater and view conditioning enter only through a lightweight decoder head with a Pionono-style latent. Drawing many rater hypotheses therefore costs one backbone pass plus cheap head evaluations, rather than $M$ full networks. Training maximizes the likelihood of each rater's contour under that rater's latent and the view condition.

The fourth component is analysis. For held-out patients we compare the predicted $R$ map with the measured disagreement reduction. We also test whether boundary segments flagged as convention-limited coincide with known clinical judgment calls, such as esophageal wall versus surrounding fat at the level of the tumor.

Baselines

The do-nothing reference is the raters themselves under conventional viewing: the measured inter-rater surface spread and the pairwise Dice among readers. SegTHOR reports an expert-level esophageal Dice of 0.80 on single-energy CT, which anchors what single-energy disagreement looks like. Our own cohort's number has to be measured.

The strongest competitor for predicting the rater distribution is a deep ensemble of deterministic 3D networks, each trained on a different rater's or bootstrap's labels. QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge reports that ensembles generally beat single-model probabilistic methods, with top teams reaching average Dice of 0.80 to 0.85 under its multi-threshold scoring. No per-task number for QUBIQ pancreas CT was found on Lacuna, and no result for any method on multi-rater spectral CT exists in the file. We would run a five-member ensemble with the same spectral input channels, crop size, augmentation, and total GPU-hour budget, and tune its hyperparameters with the same number of trials as our model. Our model beats it if it matches or exceeds the ensemble's multi-threshold Dice and its prediction of per-voxel rater variance on held-out patients, while using roughly one fifth of the inference memory. Matching the ensemble at lower cost is the bar. Exceeding it is not required for the scientific claim.

A second baseline is the same architecture trained on single-view labels only, which is the obvious stacked idea. It can estimate total disagreement but cannot produce $R$. Comparing it with the paired-view model isolates the contribution of the view-conditioned labels. A third baseline is a non-learned separability map: spectral contrast-to-noise across the boundary, computed from the material maps. It serves as a predictor of where disagreement shrinks. The key ablation removes the view variable while keeping both label sets pooled, testing whether explicit conditioning matters. A second ablation swaps spectral input for the conventional image alone, testing whether the model's $R$ prediction actually relies on spectral information.

Why not just…?

Why not just run the reader study and skip the model? The reader study is a contribution in its own right, and if only one part of this project survives it should be that one. But it answers the question only for scans that have been read twice by five experts. The model is what makes the decomposition available for a new patient at planning time, which is where it would change how a contouring system displays uncertainty. If the learned $R$ map does not predict measured reductions on held-out patients better than the separability baseline, we will report that the model adds nothing and the deliverable is the study.

Why not just compute spectral contrast-to-noise across the boundary and call high-contrast segments resolvable? This is the most serious threat, because it is cheap and physically motivated. It may fail where disagreement arises from thin structures, partial-volume effects, or missing anatomical context rather than local contrast. SegTHOR's finding that 2D networks fail in the lower chest, where landmarks vanish, hints that context matters as well as local contrast. The experiment settles it directly. We correlate both the separability map and the learned $R$ with measured per-segment disagreement reduction. If separability does as well, we adopt it and drop the learned decomposition.

Why not just train a deterministic 3D network on the spectral-view consensus (majority vote or STAPLE)? That would give a sharper contour, and it is worth including as a reference for the segmentation-accuracy check. It discards the convention-limited band, though, and that band is precisely what clinicians need to see near critical structures. The multi-rater literature shows consensus training misses systematic rater tendencies: Pionono reports Cohen's κ of 0.758 against 0.750 for the prior method on Gleason 2019, and the variational model reports 0.5372 continuous Dice against 0.5083 for Bayes dropout on QUBIQ brain growth. Those are 2D results, so we would verify on our data that the consensus model's single contour falls outside the spectral-view rater band at convention-limited segments.

Why not just use a deep ensemble and accept the memory cost? Ensembles may well win on fidelity. QUBIQ suggests they usually do. They are a baseline, not a straw man. Even an ensemble with view-conditioned training data, though, needs the paired-view labels to produce $R$. The single-pass design is offered as a cost reduction, not the core claim. If the ensemble wins clearly at equal GPU-hours, we use the ensemble inside the same decomposition and report the memory trade-off honestly.

Proposed Experiments and Decisive Checks

The first decisive check comes from the reader study. Does spectral viewing reduce inter-rater surface spread beyond the conventional test–retest control? Is the reduction concentrated on specific boundary segments, as the gradient-index result of the esophageal dosimetry study suggests, or is it uniform? A null result here, where spectral viewing changes nothing beyond rereading, would falsify the premise for these organs and should be published as such.

The second check is generalization. On held-out patients, by five-fold cross-validation, does the predicted $R$ map correlate with the measured per-segment reduction? Is that correlation higher than the correlation achieved by the contrast-to-noise separability map and by the spectral-input-removed ablation? Correlation is measured by Spearman rank across boundary segments, with patient-level bootstrap intervals.

The third check is distribution fidelity and cost. Under each view condition, we compare multi-threshold Dice against the held-out raters and per-voxel variance calibration, alongside the five-member ensemble, with peak training memory, inference memory, and wall-clock time reported. The reversible encoder's time penalty is reported in full rather than hidden.

As a transfer sanity check, the same backbone and head, with the view variable disabled, would be trained on the QUBIQ kidney and pancreas CT tasks. This shows the single-pass probabilistic model is competitive with ensembles on a public multi-rater 3D benchmark, independent of our private spectral cohort.