Two Views That Fail Differently: Spectrum-Gated Self-Training for Surgical Hyperspectral Segmentation

Self-training is a common way to adapt a medical segmentation model to a new hospital, camera or surgical situation without new annotations. The model labels the unannotated target images itself, and those guesses, called pseudo-labels, are used as training targets. The weak point is that a wrong guess made with high confidence gets reinforced on every round. Hyperspectral surgical imaging, which records a reflectance curve of about 100 narrow wavelength bands for every pixel, sharpens this problem in a specific way. Semantic Segmentation of Surgical Hyperspectral Images Under Geometric Domain Shifts shows that segmentation networks lean on spatial context, for example calling something gallbladder because it sits next to liver, instead of reading the tissue's spectrum. When linens, instruments or resections change the scene layout, their Dice score drops by about 45%. A context-reliant model is confidently wrong in exactly these scenes, so its own pseudo-labels on them are the worst possible teachers.

The proposal is to treat a pixel's own spectrum and its spatial surroundings as two separate witnesses, and to keep a pseudo-label only when both agree. One model sees only the single pixel's reflectance curve, compressed to a few dimensions. The other is a full spatial-spectral segmenter. Under a layout change, the spatial witness breaks and the spectral witness does not. Under a change in illumination or camera, the opposite tends to happen. Because their errors should be weakly correlated, agreement between them is a much cleaner filter than either model's confidence alone. The cost of making this work is dimensionality. A per-pixel spectral model with 100 inputs, trained on about 30 animals, overfits to individual subjects (the Hughes phenomenon: with few independent samples, adding input dimensions makes estimation worse). The spectral witness therefore has to be deliberately low-capacity.

Core Claim

Under geometric domain shift in surgical hyperspectral imaging, selecting pseudo-labels by agreement between a context-free, dimension-limited spectral classifier and a spatial-spectral segmenter yields more accurate pseudo-labels than single-model confidence or same-view dual-network disagreement. Self-training on those labels recovers more out-of-distribution Dice than supervised context-breaking augmentation alone.

The mechanism rests on a measurable property: on shifted target data, the errors of the two views must be less correlated than the errors of two networks that see the same input. If that property fails, the method has no basis. The proposal tests that property before testing anything else.

The intersection is open according to the file. The closest pseudo-label denoisers in medical imaging estimate confidence from things a context-crutch model gets wrong under layout change. Local-Global Pseudo-label Correction for Source-free Domain Adaptive Medical Image Segmentation corrects labels using neighbouring-pixel affinity and entropy-selected class prototypes on RGB fundus images. Its spatial-smoothness prior is the very assumption that geometric shift violates. Enhancing Pseudo Label Quality for Semi-supervised Domain-Generalized Medical Image Segmentation uses disagreement between two networks under Fourier style swaps on grayscale MRI. Both of its networks see the same spatial input, so they can share the same context error.

On the hyperspectral side, the file contains two pseudo-label methods, and neither involves domain shift. Boosting Few-Shot Hyperspectral Image Classification Using Pseudo-Label Learning assigns soft labels by spectral distance to labeled reference pixels within a single remote-sensing scene. Pixel-Superpixel Contrastive Learning and Pseudo-Label Correction for Hyperspectral Image Clustering aligns pixel and superpixel cluster assignments. The idea that a spectral-distance label is informative is borrowed from the first paper. The difference is that here the spectral view is used as an independent check on a spatial model across a shift, not as the labeler itself.

Addressing Annotation Scarcity in Hyperspectral Brain Image Segmentation with Unsupervised Domain Adaptation adapts from fundus to brain hyperspectral images with adversarial alignment and learnable band reduction, reaching 0.60 Dice. It explicitly does no self-training. That result motivates the work, since adversarial alignment alone plateaued low, but its cross-modality setting is not the one studied here.

Research Program

The primary setting is the porcine surgical hyperspectral data used by the geometric-shift paper: 100 bands spanning 500–1000 nm, L1-normalised, with organ classes. The data come from the HeiPorSPECTRAL acquisition line. The source domain is labeled, undisturbed scenes from training pigs. The target domain is the out-of-distribution scenes (isolation, resection, and tool or linen occlusion) from held-out pigs, used without labels during adaptation and scored against their held-out masks.

The spatial witness is a standard spatial-spectral segmentation network. The spectral witness is a small classifier applied to each pixel independently. Its input is a $k$-dimensional projection of the spectrum, with $k$ swept from 3 to 100. The projection is either PCA fit on source spectra or a learned linear bottleneck. Identifying chromophore fingerprints of brain tumor tissue on hyperspectral imaging using principal component analysis found that one component carries over 98% of tissue spectral variance, and HeiPorSPECTRAL attributes only 2.3% of variance to inter-subject differences. Together these suggest that a very small $k$ keeps the organ identity while discarding the subject-specific detail that drives overfitting.

On each target pixel $i$, a pseudo-label is accepted with weight

$$w_i = \mathbb{1}[\hat y^{\text{sp}}_i = \hat y^{\text{ctx}}_i]\cdot \min(p^{\text{sp}}_i, p^{\text{ctx}}_i).$$

Here $\hat y^{\text{sp}}_i$ and $\hat y^{\text{ctx}}_i$ are the predicted classes of the spectral and spatial witnesses, and $p^{\text{sp}}_i$, $p^{\text{ctx}}_i$ are their predicted probabilities for those classes. The spatial model is retrained on the weighted pseudo-labels, in the style of the uncertainty-weighted loss of Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification. The spectral witness is refit on accepted pixels only at a slow rate, so that it does not absorb the spatial model's bias. This guards against the teacher-to-student inheritance of blind spots described in Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion.

Checkpoints and $k$ are selected without target labels using the source-split imputer method of Pseudo-Label Validation for Unsupervised Domain Adaptation. That paper showed that naive source validation can select a badly overfitted checkpoint, scoring 53% against a 73% oracle on Camelyon17.

The central analysis measures, on shifted target data, the rate at which the two views are wrong on the same pixel, compared against two same-view networks. It also measures pseudo-label precision and recall per class as a function of $k$. These measurements answer the scientific question of whether view-specific errors actually decorrelate under geometric shift, and where along the dimensionality axis the spectral witness stops helping.

Baselines

The naive reference is the source-only spatial-spectral network evaluated on the out-of-distribution sets. The geometric-shift paper reports that such models lose about 45% Dice on hyperspectral input under these shifts. The file reports the drop only as a relative figure, so the absolute source-only Dice will be reproduced on the same splits.

The strongest supervised competitor is the same network trained with Organ Transplantation. This augmentation cuts labeled organs out of one scene and pastes them into another, so the model cannot rely on context. The paper reports Dice gains of up to 90% for hyperspectral input, and 283% for the pancreas under occlusion. Again, no absolute Dice is reported in the file for this setup.

The strongest self-training competitor is confidence-aware cross pseudo supervision, adapted from Enhancing Pseudo Label Quality for Semi-supervised Domain-Generalized Medical Image Segmentation. That paper reports 84.52% Dice on M&Ms at 2% labels against a prior best of 72.85%. No reported result was found on Lacuna for this method on surgical hyperspectral data. Its Fourier style swap will be applied across the spatial axes. A further competitor is a prototype-plus-local-affinity corrector adapted from Local-Global Pseudo-label Correction for Source-free Domain Adaptive Medical Image Segmentation, which reports 86.89% Dice on REFUGE to RIM-ONE-r3. No result on Lacuna was found for it on hyperspectral data. Both competitors will be started from the Organ-Transplantation model.

All methods share the same backbone, the same target pool, the same number of adaptation epochs, and the same unlabeled-target model selection procedure. Hyperparameter tuning budgets are equal.

The ablations isolate the key ingredient. The first replaces the spectral witness with a second spatial network, keeping two views but making them the same kind. The second gates on spectral confidence alone. The third removes the dimensionality bottleneck ($k=100$). The method beats the field only if it improves mean out-of-distribution Dice over Organ Transplantation plus the best adapted self-training competitor, with a margin larger than the spread across seeds and held-out-pig folds. It must also not lower in-distribution Dice.

Why not just…?

Why not just use the spectral-only classifier as the final model? If spectra are invariant to layout, a per-pixel classifier ignores context by construction. This is a real threat. The file contains no absolute per-pixel result for these out-of-distribution sets. The spectral classifier will therefore be reported as a standalone model. The proposal claims value only if the self-trained spatial model beats the spectral witness on both in-distribution and out-of-distribution data, because spatial context still helps at boundaries and with ambiguous tissue. If the spectral model alone matches the combined system, that is the finding, and it should be reported as such.

Why not just train with Organ Transplantation and stop? Organ Transplantation can only rearrange shifts that are present in labeled source data. Unlabeled target scenes, with their own occlusion patterns, are not used at all. The comparison against Organ Transplantation, and against Organ Transplantation followed by our gate, settles whether target self-training adds anything beyond augmentation.

Why not just use two networks and their disagreement, as in cross pseudo supervision? Two networks that see the same spatial input can learn the same context shortcut, so agreeing with each other proves little. The same-view ablation, together with the direct measurement of how often both views are wrong on the same pixels, tests this claim. If the two network types are not measurably less correlated, the method fails on its own premise.

Why not just use global PCA and an off-the-shelf method? SuperPCA notes that global PCA blurs local boundaries. The design here applies PCA only to the per-pixel witness, where boundaries do not matter, and leaves the full spectrum to the segmenter. The sweep over $k$ shows whether this reduction is the right place to fight dimensionality.

Proposed Experiments

The decisive first check is the error-correlation measurement on all six out-of-distribution sets. It compares the spectral-versus-spatial pair against two spatial networks.

The main result is out-of-distribution Dice per organ and per shift type, for every baseline and ablation, under leave-pigs-out folds. It is accompanied by pseudo-label precision and coverage at each self-training round, which shows whether confirmation bias is being contained.

The dimensionality curve plots pseudo-label precision and final Dice against $k$. The prediction is a peak at small $k$, with degradation as $k$ approaches 100. If the curve is flat, the curse-of-dimensionality component of the hypothesis is wrong.

A secondary transfer test runs leave-one-patient-out on the sparse human glioblastoma annotations of HELICoiD. It is reported against the 80% overall accuracy of the 3D-2D network in Hyperspectral Imaging for Glioblastoma Surgery: Improving Tumor Identification Using a Deep Spectral-Spatial Approach. This checks whether the gate helps when the shift is between patients rather than between scene layouts. In that setting the spectral view may be the one that breaks, and the prediction is a smaller gain there.