Body composition from CT, meaning how much muscle, visceral fat and subcutaneous fat a patient carries, predicts outcomes in cancer and heart disease. In principle it is cheap to measure. CT voxels are physical measurements on the Hounsfield Unit (HU) scale, which is calibrated so that air reads −1000 and water reads 0. On that scale fat falls roughly between −190 and −30 HU and muscle between +10 and +90 HU. The difficulty is location. Muscle, liver and bowel wall have overlapping densities, so a threshold cannot say which fat is visceral and which is subcutaneous. Current tools therefore train fully supervised networks on densely traced labels, and to save compute they usually measure on a handful of 2D slices. Those slice-based measurements then drift when the network meets scanners from another hospital.
We propose to split the job along the line where each piece is easiest. A 3D network learns only coarse anatomical compartments: the subcutaneous layer, the muscle compartment, the abdominal cavity, and foreign objects such as implants. These regions are smooth along the body axis, so they can be learned from very sparse slice annotations. Tissue identity inside each compartment is then assigned by HU windows, after a per-scan correction that anchors each scan's intensities to internal references (outside air and the subcutaneous fat peak). We claim this decomposition gives volumetric tissue measurements that hold up across institutions at annotation budgets far below the 1-in-5 slices used today. We also claim it beats an end-to-end network that predicts tissue classes directly from the same sparse labels. The test is whether the measured volumes, not just overlap scores, stay accurate on held-out hospitals as labels get sparser.
Core Claim
The quantity clinicians use is a tissue volume:
$$V_{t} = \Delta v \sum_{x} \mathbf{1}[\,c(x) = k_t\,]\;\mathbf{1}[\,\tilde{h}(x) \in W_t\,].$$
Here $x$ ranges over voxels, $\Delta v$ is the voxel volume, $c(x)$ is the predicted compartment, $k_t$ is the compartment that hosts tissue $t$ (for example, the abdominal cavity for visceral fat), $\tilde{h}(x)$ is the scan's HU value after per-scan correction, and $W_t$ is the tissue's HU window. The two factors are learned and supervised differently.
The compartment map is smooth in $z$ and needs anatomical context, so it is the right target for a 3D network trained on sparse slices. A compartment boundary error only changes $V_t$ where the boundary crosses tissue that falls inside $W_t$. That is why errors in the learned part should be partly absorbed rather than amplified. The HU factor needs no labels at all. Its failure mode across hospitals is a global intensity shift from tube voltage, reconstruction filter, or contrast agent, and an internal reference correction can address that directly.
The novelty is not the idea of HU-within-mask, which Comp2Comp already uses post hoc to separate intermuscular fat from muscle. It is three things together: compartment-only supervision in 3D from sparse slices, HU correction treated as the cross-site mechanism, and an evaluation that measures how far annotation can be thinned before volumetric measurements break on unseen institutions.
Research Program
The data backbone is SAROS, which supplies 900 whole-body CTs from 28 collections in The Cancer Imaging Archive. Every fifth axial slice is labeled with body regions, including muscle, subcutaneous tissue, abdominal and thoracic cavities, and breast implants, and unlabeled slices carry an "ignore" token. SAROS motivates the work twice. It showed that 1-in-5 sparse labels are enough to train strong 2D models. It also states that the sparsity left 3D modeling unexplored, so its baselines were 2D nnU-Net only. Its 28 source collections give a natural leave-collections-out split for cross-institutional testing.
We build three components.
The first is a 3D compartment model trained with a masked loss, computed only on annotated slices. We vary label density across 1-in-5, 1-in-10, 1-in-20, and a "vertebral-levels-only" regime in which roughly six slices from T12 to L5 are labeled per scan. The last regime mimics the slices Comp2Comp measures on and is the cheapest annotation a clinic would plausibly provide. To handle thick-slice anisotropy, we resample to a common grid and keep the kernels anisotropic.
The second is a single round of pseudo-labeling for unlabeled slices, filtered by physical consistency. Voxels the model assigns to the subcutaneous compartment must be predominantly fat-density. Boundaries between the subcutaneous and muscle compartments should sit on a fat-to-soft-tissue transition. Unlabeled slices whose predictions violate these checks beyond a tolerance are left out of the pseudo-label set. We stop at one round on purpose. Zhang et al. found that recursive self-training starting from a density-clustering teacher improved from 74.67% to 75.87% Dice and then fell to 74.94% at the third level as the student reinforced its own false positives. That result constrains our design: the filter exists to catch errors that intensity can expose, not to license repeated rounds.
The third is per-scan HU correction. We estimate the mode of external air and the mode of subcutaneous fat inside the predicted subcutaneous compartment, then fit an affine map that brings both modes to reference values. Fat density varies biologically, so the correction is clipped to small shifts. Its contribution is measured, not assumed. Contrast-enhanced scans, which dominate SAROS, are a known stress point because contrast raises soft-tissue HU. We report results stratified by contrast phase wherever the metadata allow.
Baselines
The naive reference point is direct global HU thresholding with a body mask and morphological cleanup, splitting fat into visceral and subcutaneous by a fixed geometric heuristic. This is what one gets for zero annotations. It bounds how much the learned compartments add. No reported result for this pipeline was found on Lacuna for SAROS, so we will run it ourselves on the same held-out collections.
The strongest existing published system is Comp2Comp. It localizes vertebrae in 3D with TotalSegmentator, segments muscle, visceral fat and subcutaneous fat on six 2D slices from T12 to L5 with densely supervised 2D U-Nets, and applies HU rules inside the masks. It reports internal L3 Dice of 0.97 for muscle, 0.96 for visceral fat and 0.97 for subcutaneous fat. On a 20-case external set it reports Dice "generally above 80% for muscle and above 85% for VAT and SAT". We will run its released models unchanged on our held-out SAROS collections, at its native vertebral levels. Beating it means lower volumetric error at matched levels and, more importantly, at the whole-abdomen level, while using only sparse slice labels.
The 2D nnU-Net baselines from SAROS, which report 0.971 Dice for muscle and 0.988 for abdominal cavity, will be retrained on the same leave-collections-out split and the same label densities. This isolates the effect of 3D context under identical supervision.
The most dangerous competitor has no published number: a plain 3D nnU-Net-style model trained with the ignore-masked loss on sparse slices, predicting tissue classes directly (end-to-end) instead of compartments followed by HU. No reported result was found on Lacuna for 3D training on SAROS sparse labels. We give it the same data, the same architecture family, the same GPU budget and an identical hyperparameter search. Beating it requires a statistically significant reduction in held-out-site volume error at 1-in-10 and sparser.
For the generative route, Zuo et al. report a subcutaneous fat-to-muscle ratio error of 15.2% for synthesized 3D volumes versus 23.3% for 2D, but a visceral fat error of 26.3% versus 24.2% for 2D. We will compare against their reported numbers in the vertebral-levels-only regime and, if code is available, rerun them.
Ablations remove one ingredient at a time: the HU correction, the physical pseudo-label filter, and the decomposition itself (the end-to-end variant above).
Why not just…?
Why not just threshold HU values in 3D and skip learning? Thresholding answers "is this fat" but not "is this visceral fat." Muscle and organ parenchyma share densities, so masks leak where the abdominal wall touches bowel or liver. Implants also mimic soft tissue, which is why SAROS labels them separately and still reaches only 0.585 Dice on them. In Zhang et al., density clustering alone reached 74.67% Dice and needed a learned student to improve. The naive baseline above quantifies this gap on our data.
Why not just keep 2D slice networks, as Comp2Comp does? Single-slice measurements vary with where along the body the slice lands. Zuo et al. attribute a 23.3% ratio error to exactly this. Independent 2D predictions also break continuity across slices, as Roth et al. observed for pancreas. The 2D-versus-3D comparison at matched label density in our program settles whether this matters for whole-abdomen volumes.
Why not just train a 3D network end-to-end on the sparse labels? This is the real threat. If a 3D tissue segmenter trained with the ignore loss matches the decomposed model at every label density and on every held-out site, then the decomposition adds nothing. The contribution would shrink to the annotation-density curve and the HU correction. The decisive ablation is designed to show this outcome if it is true. We expect the gap to open at very sparse labels and on sites with shifted HU, because there the end-to-end model must learn intensity boundaries from few examples, while the decomposed model gets them from physics.
Why not just impute missing slices with a diffusion model? Zuo et al. show that this helps smooth outer boundaries but worsens visceral fat. Visceral fat is precisely the convoluted internal structure where synthesized intensities are least trustworthy. Our approach never synthesizes intensities. It reads real HU values inside a learned region.
Why not just pool more sites and use a bigger model? A-Eval found that joint multi-dataset training transfers well, reaching 88.90% DSC on unseen BTCV. It also found that scaling from STU-Net-L to STU-Net-H stopped helping, and that pooling presupposes dense labels everywhere. We use pooled training too. The question is what happens to the annotation cost.
Proposed Experiments
The primary split holds out whole SAROS collections, grouped where possible by scanner vendor, so that test scanners are unseen. On held-out collections, evaluation uses their native 1-in-5 annotated slices. We report compartment Dice and Normalized Surface Dice, but the headline metric is the relative error of fat and muscle area per slice, summed into volumes over the labeled slices of the L1–L5 span. Because SAROS has no dense ground truth, we will densely annotate a small external set of about 20 abdominal scans across compartments to check volume error on intermediate slices. This is the only new annotation the study needs.
Three decisive checks determine success. First, the error-versus-label-density curve: the decomposed 3D model should degrade more slowly than both the end-to-end 3D model and the 2D baselines from 1-in-5 down to vertebral-levels-only. Second, the site-shift test: on held-out collections we inject controlled global HU offsets of ±10 to ±30 HU, which mimic calibration drift, and stratify results by contrast phase. HU correction should flatten the error of fixed windows. If it does not, we drop it and report that physics-anchoring failed for this reason. Third, the implant subset: scans with breast implants should show no implant-driven inflation of soft-tissue volume under the decomposed model, compared with thresholding. If the end-to-end 3D model is statistically indistinguishable on all three checks, we will report that sparse 3D supervision alone is sufficient, which is itself a useful answer for clinics deciding how much to annotate.
References
- Comp2Comp: Open-Source Body Composition Assessment on Computed Tomography
- SAROS: A dataset for whole-body region and organ segmentation in CT imaging
- Self-learning to detect and segment cysts in lung CT images without manual annotation
- Robust Body Composition Analysis by Generating 3D CT Volumes from Limited 2D Slices
- Deep convolutional networks for pancreas segmentation in CT imaging
- A-Eval: A Benchmark for Cross-Dataset Evaluation of Abdominal Multi-Organ Segmentation