Held-Out Scribbles as Referees for Co-Training Across Scanners

When a segmentation model trained at one hospital is deployed at another, its accuracy can collapse. Scanners, sequences and patient populations all differ, and the model's own confidence stops being a trustworthy guide to where it is wrong. The usual fix is to label some data at the new site, but contouring 3D organs takes radiologists hours per scan. A cheap alternative is the scribble: a few quick strokes inside each structure. This proposal asks how far scribbles alone, at both the original and the new site, can carry a model across a scanner or sequence change. It also asks whether two models teaching each other can do better than one model trained on the scribbles directly.

The idea is simple. Split each scan's scribble strokes into two disjoint halves and give each half to a different network. Each network has never seen its partner's strokes, so its accuracy on those strokes is an honest, label-grounded estimate of how reliable it is at the new site. The estimate can be broken down by structure and by distance from the nearest stroke. We then use these measured reliabilities, rather than the networks' self-reported confidence, to decide which network's prediction becomes the training target for the other wherever the two disagree. Under domain shift, self-reported confidence is the quantity that is known to break.

Core Claim

Collaborative pseudo-labeling means two networks trained in parallel, each learning from the other's predictions on unlabeled voxels so that neither reinforces only its own mistakes. All existing versions resolve disagreements with signals internal to the models. DMT weights or zeroes losses by relative confidence. Alternate Diverse Teaching (AD-MT) picks the lowest-entropy teacher at conflicting pixels. ScribbleVS picks whichever of student or teacher scores better on the scribbles, but those are scribbles both networks were trained on.

Under scanner shift these internal signals become unreliable. Improved post-hoc probability calibration for out-of-domain MRI segmentation motivates this point: it shows that calibration error is large after a bSSFP-to-LGE change (15.37% ECE even after its correction). Our claim is that strokes withheld from one peer give an external referee that remains unbiased under shift. Arbitrating peer disagreement with this referee should beat both confidence-based arbitration and a single well-tuned scribble-supervised network, when source and target sites both have only scribbles.

Concretely, let $r_A(c, d)$ be the fraction of network B's stroke voxels of class $c$, lying in distance band $d$ from A's own nearest stroke, that network A labels correctly. When A proposes label $c$ for voxel $v$ to train B, the loss on that voxel is weighted as

$$w_{A\to B}(v) = r_A(c, d(v)) \cdot \mathbb{1}\left[r_A(c, d(v)) \ge r_B(\hat{c}_B(v), d(v))\right],$$

where $\hat{c}_B(v)$ is B's own prediction and $d(v)$ is the voxel's distance band. In words, a peer's label is used only where that peer has been measured to be at least as accurate as the receiver. It is weighted by that measured accuracy. Both $r$ tables are re-estimated every epoch on target-site strokes.

Band-wise estimates matter most near boundaries. Scribble methods are weakest there: ScribbleVS masks boundaries out, and cross-image matching reports poor 95th-percentile Hausdorff distance. ScribbleBench includes border scribbles, which supply held-out evidence in exactly those bands.

Research Program

The setting is scribble-to-scribble adaptation. Source-site scans carry scribbles only, and the target site has scribbles on a small number $k$ of scans plus unlabeled scans. This is the gap the file identifies. ACT and UMCT co-train across domains but need dense source masks, and ACT also needs dense target seed subjects. Dual-supervised Asymmetric Co-training (DAC) needs dense source labels and works on 2D slices. Scribble-based Domain Adaptation via Co-segmentation uses target scribbles but relies on 150 densely labeled source scans and a single network. We take the collaborative structure from the first group and the target-scribble regime from the last. The new element is replacing their confidence or entropy arbitration with held-out-stroke reliability.

The backbone is deliberately the simplest strong one: nnU-Net with a partial loss, meaning cross-entropy computed only on scribbled voxels. Embarrassingly Simple Scribble Supervision and Revisiting 3D Medical Scribble Supervision show this baseline generalizes across seven 3D datasets, while specialized cardiac-tuned methods collapse. Those papers constrain the design: any added machinery must be shown not to be a cardiac quirk, so we avoid shape-specific heuristics.

Stroke splitting serves two purposes. It makes the two peers diverse without architectural tricks, and the literature notes that diversity is what prevents "two identical wrongs." It also produces the referee data. We build in three further pieces. The first is a warm-up phase in which each peer trains on its own strokes only, addressing cold-start instability. The second is per-epoch re-estimation of $r$ on target strokes only, so reliability reflects the deployment site. The third is a collapse monitor: if held-out-stroke accuracy of the two peers converges while their disagreement on unlabeled voxels drops to near zero, we flag consensus collapse rather than progress.

A secondary analysis asks whether the reliability tables predict true dense-label accuracy. Every benchmark we use has dense target masks available for evaluation only, so this can be tested directly.

Baselines

The naive reference is a source-only nnU-Net trained with partial loss on source scribbles and applied unchanged to the target site. This measures how large the scribble-regime shift gap actually is. AMOS reports a drop of over 20% on unseen MRI scanners for densely trained models. No reported result was found on Lacuna for scribble-trained models under that shift.

The strongest competitor is nnU-Net with partial loss trained on pooled source scribbles plus the $k$ target scribbled scans. On in-domain ScribbleBench this recipe reaches 0.813 mean Dice against 0.856 for dense supervision, as reported in Revisiting 3D Medical Scribble Supervision. It is the competitor most likely to make the proposal unnecessary. To keep compute equal, because our method trains two networks, it gets the same GPU hours: we report it both as one network trained twice as long and as an ensemble of two independently trained copies. Its hyperparameters receive the same tuning budget, using the same nnU-Net automatic configuration.

The third baseline is single-model self-training. The pooled model labels the unlabeled target scans, confident voxels above a threshold tuned on a held-out target split become targets, and the model retrains. The fourth is ScribbleVS-style competitive selection on a mean teacher, ported to the same backbone.

Where dense source labels are allowed, we also report the methods built for that richer setting as upper references. On Vestibular Schwannoma T1-c to T2, the co-segmentation method reports 83.4% target Dice with 150 dense source scans and 30 scribbled target scans Scribble-based Domain Adaptation via Co-segmentation. On BraTS18 T2 to T1, ACT reports 75.7% Dice with 5 densely labeled target subjects ACT.

Three ablations isolate the key ingredient. The first trains both peers on all strokes and estimates reliability on training strokes; this removes the held-out property. The second keeps the stroke split but arbitrates by minimum entropy as in AD-MT. The third keeps the split and the two peers but exchanges no pseudo-labels and simply averages them.

Beating the pooled baseline means a statistically significant gain in target Dice and a reduction in target 95HD, using a paired test over target test scans across three seeds, at matched compute, for at least two of the three shift settings. The held-out-split ablation must account for most of that gain. If only the no-exchange ensemble matches us, the contribution is ensembling, not collaboration, and we will say so.

Why not just…?

Why not just pool all the scribbles and train one nnU-Net? This is the real threat. The ScribbleBench results show that simple partial-loss training is hard to beat and that clever pseudo-labeling often hurts. Our first experiment therefore measures the target-domain gap left by the pooled model as $k$ varies. If that gap is within a point or two of dense supervision on the target, the proposal reduces to "use nnU-Net," and we will report that as the finding. Our bet rests on the AMOS evidence that unseen scanners cause large drops even with dense labels. A handful of target scribbles is unlikely to fix boundary errors that scribbles do not cover.

Why not just self-train one model with a confidence threshold? Under shift, confidence is miscalibrated in exactly the direction that hurts: overconfident errors on the new scanner, as the out-of-domain calibration paper documents. A threshold tuned on source behaves differently on target. The entropy-arbitration and training-stroke ablations test this directly. If held-out reliability does not beat them, the core claim fails.

Why not just use the scribbles as a validation set for model selection, without co-training? Held-out strokes could simply pick checkpoints or ensemble weights for the pooled model. This is cheaper, and we include it as a variant of the strongest baseline. The difference is that our use is spatially resolved and feeds back into training: reliability per class and per boundary band decides which voxels supervise which network. If global selection matches voxel-wise arbitration, the extra machinery is not justified.

Proposed Experiments

The first setting is AMOS. Source is CT, or CT from a subset of scanner vendors. Target is MRI, or held-out vendors. We use ScribbleBench's generated interior and border scribbles, with $k \in \{0, 5, 10, 30\}$ scribbled target scans and the remaining target scans unlabeled except for a dense test split. The second setting is MS-CMRSeg, bSSFP to LGE, which has a scribble release for LGE. The third is BraTS cross-sequence, T2 to T1 and FLAIR, using ScribbleBench-style scribbles. Vestibular Schwannoma T1-c to T2 serves as a check against the dense-source co-segmentation number. Metrics are per-structure Dice, 95HD, and expected calibration error on the target test split, plus scribbled-scan count as the annotation cost axis.

The decisive checks are three. The first asks whether the pooled nnU-Net leaves a meaningful target gap at all; if not, we stop. The second asks whether $r_A(c, d)$ estimated from held-out strokes correlates with true dense-label accuracy in the same class and band on target scans. We repeat this with interior-only strokes to test whether border scribbles are needed for the boundary bands. If the correlation is weak, the referee is not trustworthy and the method cannot work for the stated reason. The third asks whether, at matched compute, the full method beats the pooled and ensembled baselines and the training-stroke ablation on target Dice and 95HD. The gain should be concentrated in boundary bands, which is what the mechanism predicts and what an ensembling effect alone would not produce.