Hospitals rarely label every organ in a scan. One dataset marks the liver, another the kidneys, a third the pancreas. Everything else is left blank, even though it is full of other organs. The accepted fix, the marginal loss, avoids punishing the model for predicting an unlabeled organ inside blank regions. The catch is that it gives no signal about which unlabeled organ a blank voxel belongs to. Current methods fill that gap with entropy minimization, which simply pushes the model to be confident. With thousands of scans this works. With five to ten scans per labeling scheme, the regime clinical groups actually face, confidence can be confidently wrong, or collapse onto whichever missing organ is most common.
We propose a different filler signal built from multi-scale convolution. The segmentation network gets two prediction branches that see the scan at different scales. A small-kernel branch reads local texture. A dilated branch, a convolution with gaps between its weights so it sees a wide area at no extra parameter cost, reads anatomical layout. Inside blank regions, the two branches must agree on which of the scan's missing organs each voxel is. Texture and layout make different mistakes: the pancreas and duodenum look alike locally but sit in different places. Their agreement is therefore a more trustworthy self-supervision signal than either branch's confidence alone. In two sentences: replace "be confident on unlabeled voxels" with "let a texture view and a layout view check each other on unlabeled voxels." Test whether this helps specifically when labels are both partial and scarce.
Core Claim
Under disjoint partial labels with small cohorts, the main error is confusion among classes that were jointly unannotated in a training scan. We call this pooled-class confusion, because the marginal loss lumps those classes together.
We claim that cross-scale agreement reduces this confusion more than entropy minimization does, at matched parameters and memory. We also claim the gain comes from the difference in receptive field between the branches, not from having two branches. The second part is falsifiable with an ablation where both branches have the same scale.
Formally, let a training scan come from a site whose annotated organ set is $L$. A voxel $i$ that is annotated has compatible set $C_i=\{y_i\}$. An unannotated voxel has $C_i=U=\{\text{background}\}\cup(\text{all organs not in } L)$. With $p^{s}_i$ and $p^{\ell}_i$ the small- and large-receptive-field branch distributions over classes, and $\tilde p_i = \tfrac12(p^{s}_i+p^{\ell}_i)$ their average, the objective is
$$ \mathcal{L} = -\sum_{i}\log\sum_{c\in C_i}\tilde p_i(c)\;+\;\lambda\sum_{i:\,C_i=U}\mathrm{JS}\!\left(p^{s}_i|_U\,\big\|\,p^{\ell}_i|_U\right), $$
where the first term is the marginal loss and $\lambda$ weights the agreement term. In the agreement term, $p|_U$ is a branch distribution renormalized over the compatible set $U$, and JS is the Jensen–Shannon divergence. JS is bounded, so it cannot reward collapse the way entropy minimization can: two branches agreeing on a uniform distribution is not penalized. To guard against agreement on a trivial answer, each branch also receives the marginal loss separately.
Prior Work
Three groups of prior work set up the problem.
The partial-label method comes from sequence tagging. Kashima et al., Marcheggiani et al., Jie et al., and Marginal Likelihood Training of BiLSTM-CRF for Biomedical Named Entity Recognition from Disjoint Label Sets maximize the probability of all label assignments consistent with the partial annotation, instead of treating blanks as negatives. We reuse that compatible-set loss per voxel.
Two medical papers motivate the gap.
- Versatile Medical Image Segmentation Learned from Multi-Source Datasets via Model Self-Disambiguation pairs this loss with entropy minimization. It reaches 88.7% Dice on 16 structures, but on 2,960 scans. It never tests small cohorts, and it lists entropy-driven collapse as an open risk.
- ConDistFL uses a marginal loss plus distillation from a global teacher in background regions. That still requires a teacher trained on enough data to be useful.
Our proposal keeps the marginal loss and replaces both entropy and the teacher with an internal two-scale check.
Multi-scale convolution supplies the architecture.
- PSConv cycles dilation rates across channels inside one layer at zero parameter cost. We extend it to 3D so the wide branch adds no memory beyond activations, which matters because 3D medical training is already stuck at batch size 1.
- CFPNet shows that channel-split dilations with stepwise fusion avoid gridding artifacts, the checkerboard holes of a single large dilation rate.
- Dual-Kernel Adapter is the closest prior work. It found that standard adapters suffer a shrinking effective receptive field below 1% data. Its parallel 51×51 and 5×5 depthwise branches fix this, giving 23.02% mIoU on BRATS at 1.25% data. But it uses fully labeled data and simply sums the branches. It neither exploits disagreement between branches as a training signal nor handles missing classes.
Label-Efficient Hybrid-Supervised Learning for Medical Image Segmentation uses two co-regularized networks, but against coarse geometric labels within a shared class schema. Its same-architecture co-training is exactly the competitor our same-scale ablation must beat.
One constraint shapes the design. AMOS finds a plain 3D U-Net (89.04% mDice) beats Swin-UNetr (86.32%) on CT. We therefore build on a 3D U-Net rather than a heavier backbone, and add only the two output branches.
Research Program
Benchmark. AMOS is fully labeled across 15 organs, so it lets us simulate disjoint partial labels while scoring against complete ground truth. We split AMOS CT training scans into five simulated sites. Each site keeps labels for a disjoint subset of three organs, and the rest are blanked. We vary the cohort at 2, 5, and 10 scans per site. As a natural partial-label setting, we also centralize the MSD liver, MSD pancreas, MSD spleen, and KiTS19 kidney split used by ConDistFL, subsampled to the same sizes.
Model. The model is a shared 3D U-Net encoder–decoder. Its last decoder stage forks into two heads:
- a small-kernel head with 3×3×3 convolutions;
- a 3D poly-scale dilated head with cyclic rates 1, 2, 4, 8 across channels and CFPNet-style hierarchical fusion.
Parameter counts are matched by construction.
Analysis. Beyond Dice, we measure three things:
- pooled-class confusion: on voxels whose true organ was unannotated at their training site, the error rate among organs in the same pooled set;
- collapse rate: the fraction of unannotated voxels assigned to the single most frequent missing class;
- effective receptive field, computed by input-gradient maps as in Dual-Kernel Adapter.
The last one lets us check whether the dilated head actually reaches neighboring labeled organs.
Baselines
The naive reference is background imputation. Here every blank voxel is labeled background with standard cross-entropy, as FedAvg-style local training does. In ConDistFL's federated setting this gave 0.7281 Dice against 0.7700 for marginal handling. We expect it to be clearly worst. It anchors how much any partial-label method buys.
Marginal loss alone, single head. This is ConDistFL's local loss without distillation, run centralized. It isolates what the filler signal adds.
Strongest existing method: marginal loss plus entropy minimization, as in Self-Disambiguation. It reports 88.7% average Dice versus 83.5% for DoDNet, on its 2,960-scan pool. That number is not comparable to our cohorts. We found no reported result on Lacuna for any partial-label method on AMOS with 10 or fewer scans per site, so we will produce that number ourselves. We will reimplement the method on our exact backbone and splits, tuning its entropy weight on the same validation budget we use for $\lambda$: 12 trials each, selected on held-out partially labeled scans with the marginal loss as the criterion, never on full ground truth.
Ablations isolating the key ingredient. - Same-scale two-head agreement: both heads use 3×3×3 kernels with different initialization. This is the DCR-style co-training control. - Two-scale heads, summed, with entropy instead of agreement: the Dual-Kernel-Adapter design ported to partial labels. - Dense large-kernel head matched to the dilated head's receptive field.
Upper bound. A fully supervised U-Net trained on the same scans with complete labels. The AMOS full-data 3D U-Net figure of 89.04% mDice is a distant ceiling.
What counts as beating the strongest baseline. Our method must lower pooled-class confusion and raise mean Dice over marginal-plus-entropy at 5 and 10 scans per site. It must also beat the same-scale agreement control. Margins must exceed the spread across three random site partitions.
Why not just…?
Why not just keep entropy minimization and tune it carefully? This is the main competitor, and it may hold up at 10 scans per site. Our bet is that it fails at 2–5 scans, where confident errors have nothing to correct them. The collapse-rate measurement makes this visible directly. If tuned entropy matches our method on pooled-class confusion at every cohort size, the claim is wrong and we will report that.
Why not just co-train two identical heads? Agreement between same-scale views is a known regularizer, and it may capture most of the gain. The same-scale ablation is built to settle this. Our hypothesis requires the two-scale version to win specifically on organ pairs that look alike locally but are spatially distinct, such as pancreas versus duodenum, or adrenal glands versus surrounding fat. A uniform gain would point to ensembling, not scale.
Why not just rely on the U-Net's own depth for context? A deep U-Net's theoretical receptive field already spans a patch. Dual-Kernel Adapter shows that the effective field shrinks under scarce data. Our receptive-field maps test whether this happens in the 3D partial-label case too. If the plain U-Net's effective field already reaches neighboring anchor organs and the dilated head adds nothing, the architectural half of the proposal collapses into a loss-only contribution.
Why not just distill from a teacher, as ConDistFL does? With 2–10 scans per site there is no strong teacher to distill from. Training one on the pooled partial data is circular, since it faces the same pooled-class ambiguity. We include a teacher variant trained on all sites as a secondary comparison.
Proposed Experiments
Main grid. Five methods (imputation, marginal, marginal+entropy, same-scale agreement, cross-scale agreement), plus the summed-two-scale and dense-kernel ablations. Each runs at three cohort sizes on three random site partitions of AMOS CT, and again on the subsampled MSD/KiTS split. Metrics are per-organ Dice reported separately for organs labeled versus unlabeled at each site, pooled-class confusion, collapse rate, and 95th-percentile surface distance.
Robustness check. Evaluate on AMOS MRI from scanners unseen in training, where models trained from scratch were reported to drop over 20%. This asks whether a layout-based view transfers better than a texture-based one.
Decisive check. The proposal stands only if three conditions hold together:
- cross-scale agreement beats same-scale agreement on pooled-class confusion;
- the gain concentrates on spatially separable but texturally similar organ pairs;
- the gain grows as cohorts shrink.
If the gain is uniform across organ pairs, or vanishes against tuned entropy, we conclude that scale diversity is not the operative ingredient.
References
- Training Conditional Random Fields Using Incomplete Annotations
- An Experimental Comparison of Active Learning Strategies for Partially Labeled Sequences
- Better Modeling of Incomplete Annotations for Named Entity Recognition
- Marginal Likelihood Training of BiLSTM-CRF for Biomedical Named Entity Recognition from Disjoint Label Sets
- Versatile Medical Image Segmentation Learned from Multi-Source Datasets via Model Self-Disambiguation
- ConDistFL: Conditional Distillation for Federated Learning from Partially Annotated Data
- PSConv: Squeezing Feature Pyramid into One Compact Poly-Scale Convolutional Layer
- CFPNET: Channel-Wise Feature Pyramid For Real-Time Semantic Segmentation
- Dual-Kernel Adapter: Expanding Spatial Horizons for Data-Constrained Medical Image Analysis
- Label-Efficient Hybrid-Supervised Learning for Medical Image Segmentation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation