Spectral Evidence First: Gated Context for Rare Organs in Surgical Hyperspectral Segmentation

A hyperspectral camera records, for every pixel, a reflectance curve across about a hundred narrow wavelength bands. That curve is a physical fingerprint of the tissue. In principle, it should let a segmentation model recognize a pancreas from its own pixels. In practice, deep models trained on surgical hyperspectral images learn a shortcut. They label the gallbladder "gallbladder" because it sits next to the liver. When a scene changes geometry, because an organ is resected, isolated on linens, or covered by a hand or instrument, performance falls by roughly 45% in Dice score (DSC, the overlap between predicted and true masks) (Semantic Segmentation of Surgical Hyperspectral Images Under Geometric Domain Shifts). Meanwhile, medical transformers show that long-range context is exactly what resolves ambiguous, small, low-contrast structures (UNETR: Transformers for 3D Medical Image Segmentation). So context helps and hurts at the same time, and the literature has no principled rule for when to trust it.

We propose a concrete rule and a testable explanation for why the shortcut forms. The explanation is that the shortcut is largely a class-imbalance effect. Rare, small organs contribute few pixels, so their spectral evidence is undertrained, and the network compensates with neighborhood topology. The rule is architectural. A per-pixel spectral classifier, which never mixes information across space, is trained with an imbalance-aware loss so that rare organs get a strong spectral decision of their own. A global context stream is then allowed to add evidence only in proportion to how uncertain that spectral decision is. The model should use context to break ties, not to overrule a confident fingerprint.

Core Claim

The claim has two parts. First, a model's reliance on spatial context is concentrated in minority classes. We expect a class's drop in DSC under geometric shift to correlate negatively with its training pixel frequency and with the accuracy of a pixel-only spectral classifier on that class. Second, three ingredients together should recover most of the out-of-distribution loss without sacrificing in-distribution accuracy: a spectral-only head trained with a differentiable F1-style loss, context that enters through a gate driven by that head's uncertainty, and training in which context is randomly corrupted. Written as one equation, the per-pixel class scores are

$$z_i = s(x_i) + g\big(H[s(x_i)]\big)\, c(X)_i ,$$

where $x_i$ is the reflectance curve at pixel $i$, $s$ is the spectral head applied pixel by pixel, $X$ is the whole cube, $c$ is the context stream's per-pixel output, $H$ is the entropy of the spectral head's softmax, and $g$ is a small learned monotone function taking values in $[0,1]$. Because $g$ is monotone in $H$, a confident spectral call cannot be overturned by context. That is the property we want under resection or occlusion.

Where This Sits in Prior Work

The work that motivates us most directly is the pair of geometric-shift papers from the Heidelberg surgical group (Semantic Segmentation of Surgical Hyperspectral Images Under Geometric Domain Shifts; Handling Geometric Domain Shifts in Semantic Segmentation of Surgical RGB and Hyperspectral Images). They diagnosed the context shortcut and fixed it with Organ Transplantation, which pastes organ masks from one scene into another during training. That improved out-of-distribution DSC by up to 90% for hyperspectral input and by 283% for the occluded pancreas. Organ Transplantation is also our strongest competitor.

Our proposal differs in three ways. It changes the model rather than only the data. It makes the trust in context an explicit, inspectable quantity. And it treats the rarity of organs such as the pancreas as the cause to be addressed, whereas those papers observe the pancreas result without framing it as imbalance.

The claim that spectra alone should suffice is supported by HeiPorSPECTRAL. That dataset reports that organs form distinct spectral clusters and that inter-subject variability explains only 2.3% of spectral variance.

The imbalance-aware loss comes from Imbalance-aware loss functions improve medical image classification. That paper builds differentiable "soft" versions of F1 and Matthews correlation from class probabilities. It combines them with cross-entropy and reports up to 12.7% relative gain in balanced accuracy on glioma MRI. It also warns that the soft losses are unstable without cross-entropy, so we keep the combined form. That paper works on image-level labels in MRI and fundus photographs; we apply the loss pixel-wise, and only to the spectral head.

For the context stream we borrow spectral-wise attention from MST++. There, each band's feature map is a token, which gives a global spatial receptive field at cost linear in image size. That matters because our 480×640 cubes make patch-token attention expensive.

Two findings constrain the design. Photometric augmentation corrupts physical reflectance values (Data Augmentation Approaches for Satellite Imagery), and projecting spectra onto a few components can suppress the subtle variation needed for rare targets (Band selection-based collaborative representation for anomaly detection in hyperspectral images). The spectral head therefore sees the full L1-normalized curve with no color jitter and no PCA.

Finally, the pixel-wise brain-tumor unmixing work shows the opposite extreme: strong per-pixel spectral models that discard space entirely. Deep Learning-Based Correction and Unmixing of Hyperspectral Images for Brain Tumor Surgery names spatial context as needed future work. Our gate is one answer to how context could be added without letting it dominate.

Research Program

The first phase is diagnostic. On the surgical geometric shift benchmark, we train a standard segmentation network and a pixel-only spectral classifier. For each of the 19 classes we record training pixel frequency, in-distribution DSC, and out-of-distribution DSC across the isolation, resection, and occlusion settings. We also measure context sensitivity directly: we replace every pixel outside an organ's mask with pixels from a different scene and record how often the organ's predicted label changes. If context sensitivity and DSC drop are not concentrated in low-frequency classes, the imbalance explanation is wrong. The architecture would then have to stand on the gate alone, and we would say so.

The second phase builds the model. The spectral head is a small one-dimensional network over bands, applied per pixel. It is trained with cross-entropy plus the soft-F1 loss averaged over classes. The context stream is a spectral-wise transformer over the cube, and the gate follows the equation above. During training, context is corrupted with some probability: the cube seen by the context stream is replaced by an Organ Transplantation-style composite, while the spectral head sees the true pixels. This teaches the gate that context can lie.

The third phase analyzes the trained model. We check whether the learned gate opens mainly where it should: at organ boundaries, under specular highlights, and on classes whose spectra overlap. We also check whether the gate's value predicts error under shift, which would make it usable as an intraoperative warning signal.

Baselines

The naive reference point is the standard segmentation network trained with cross-entropy on unaltered scenes, as in the geometric-shift papers. It sets the scale of the out-of-distribution collapse.

The second baseline is the pixel-only spectral head alone, trained with cross-entropy plus soft-F1. It is immune to the context shortcut by construction. It shows how much in-distribution accuracy context actually buys, and it bounds what the gate can gain.

The strongest existing method is Organ Transplantation (Handling Geometric Domain Shifts in Semantic Segmentation of Surgical RGB and Hyperspectral Images). It is the only published method on this exact benchmark, and it already fixes much of the problem. The file reports it only in relative terms: up to +90% out-of-distribution DSC for hyperspectral input over the baseline, and +283% for the pancreas under occlusion. No absolute DSC is reported on Lacuna for Organ Transplantation on this benchmark, so we will reproduce it and report our reproduced absolute numbers alongside the published relative gains.

We run Organ Transplantation in two forms, both with the same backbone family, the same compute, and the same hyperparameter search budget as our model. The first uses its original loss. The second, stronger form adds our imbalance-aware loss, so that any win cannot be attributed to the loss alone. A transformer baseline, a SwinUNETR-style network as used in TraBS, is trained with Organ Transplantation to check that the gate, not attention, is what matters.

Ablations isolate the key ingredient. They are: the gate replaced by a constant (plain sum of the two streams); the gate fed random noise instead of spectral entropy; no context corruption; and cross-entropy only on the spectral head.

Beating Organ Transplantation means higher macro out-of-distribution DSC averaged over the six shift settings and a higher out-of-distribution DSC for the rarest five classes. It also means in-distribution macro DSC within one standard deviation of Organ Transplantation, where the standard deviation is taken across pig-level cross-validation folds.

Why not just…?

Why not just use Organ Transplantation with a class-weighted or soft-F1 loss? This is the real threat. Organ Transplantation already decouples organs from their neighbors, and adding an imbalance loss may close the rare-class gap. That is why this combination is our headline baseline rather than a straw man. If it matches our model, the honest conclusion is that data plus loss suffices and the gate adds only interpretability, which we would then evaluate on its own through the gate-predicts-error test. Our expectation that it falls short rests on one observation: augmentation cannot make context trustworthy under shifts it never samples. The occlusion by hands and instruments is such a case, since hands and instruments are not organs that could be transplanted.

Why not just use a pixel-wise spectral classifier and ignore context? HeiPorSPECTRAL's clean spectral clustering suggests this could work well. The pixel-only baseline measures it directly. We expect it to lose on classes with overlapping spectra and at boundaries, which is the regime where UNETR's gains concentrated on the gallbladder and adrenal glands (UNETR). If it does not lose there, the context stream is unnecessary and we will report that.

Why not just use a well-tuned CNN such as nnU-Net with focal loss? Such a network still has a spatial receptive field that can learn neighbor topology. Nothing in focal loss stops the shortcut. Cross-entropy combined with soft metric losses outperformed standard losses on imbalanced medical data (Imbalance-aware loss functions improve medical image classification). We include this network in the diagnostic phase as a second standard model, so its context sensitivity is measured, not assumed.

Decisive Checks

The primary data are the 600-cube, 19-class surgical geometric shift benchmark with its six out-of-distribution settings. Splits are pig-level so that no animal appears in both training and test. HeiPorSPECTRAL (5,758 cubes, 20 organs, 11 pigs) serves for additional spectral-head training and a second in-distribution check.

Metrics are per-class and macro DSC in and out of distribution, rare-class DSC, the context-swap flip rate, and the area under the curve of gate value as a predictor of pixel error.

Three results would settle the proposal. The first is whether out-of-distribution DSC drop correlates with class rarity in the standard network; if it does not, the imbalance explanation fails. The second is whether our model beats Organ Transplantation combined with the soft-F1 loss on rare-class out-of-distribution DSC under occlusion, the setting augmentation cannot sample. The third is whether the constant-gate ablation loses that advantage. That last comparison isolates the contribution of letting spectral certainty decide how much context to trust.