Many medical segmentation systems work in two stages. A first network makes a rough guess about where a lesion is, and a second network refines the edge inside the region the first one picked. This fails in two related ways when the lesion blends into surrounding tissue. If the first stage misses the lesion, the second stage never looks at it. If the first stage finds it, the second stage usually studies a "boundary band" of fixed width carved around the rough mask, and that width is wrong for thin, branching, or lobulated shapes. We propose a different way to connect the two stages. Build a small graph over image regions, using the network's own features to measure how similar regions are. Split that graph into several segments with a classical technique called spectral rotation, which finds the rotation that best aligns a continuous embedding with discrete segment labels. Then use the rough first-stage prediction only to tell the method which segments are confidently lesion and which are confidently background.
This matters because the alignment gives two quantities that current cascades lack. The first is a per-region misfit score: how far each region's rotated embedding sits from its assigned label. That score marks the ambiguous margin at whatever width and shape the data implies, with no fixed structuring element. The second is a set of segments that no confident label claims. These are places where the graph sees a coherent structure that the first stage ignored, which makes them candidate missed lesions. The core claim is that treating the coarse-to-fine interface as a partially anchored discretization problem, rather than a crop or a morphological band, both locates the ambiguous boundary better and partially lifts the rule that final recall cannot exceed first-stage recall.
What exists and what is missing
Spectral rotation comes from Multiclass Spectral Clustering. That paper showed that the eigenvectors of a graph are only defined up to an arbitrary rotation. It recovered discrete labels by alternating two steps. The first is a closed-form rotation step: take the singular value decomposition (SVD) of a small $k\times k$ matrix, a standard factorization, and read off the best rotation. The second is a step that assigns each row to its largest entry. Spectral Rotation versus K-Means in Spectral Clustering showed that this optimizes the graph cut better than running K-means on the eigenvectors, for example raising COIL20 clustering accuracy from 61.2% to 67.5%. Both papers note a limitation that constrains our design: rotation cannot repair a graph that already links lesion to background. We therefore build the graph from learned features rather than raw intensities. Unified formulations such as A Unified Framework for Discrete Spectral Clustering and Unified Spectral Clustering With Optimal Graph fold graph learning and rotation into one objective. They remain unsupervised, flat, and quadratic in the number of nodes, so they are unusable on dense images. None of this work uses partial label anchors, and none reads the alignment misfit as a boundary signal.
On the cascade side, RSTN replaces hard crops with soft saliency weighting across recurrent passes. PDSL runs detection and segmentation in parallel so neither branch filters the other. RecycleNet feeds latent features back into earlier layers. These works motivate our proposal, since each documents cascade failures. They also bound its ambition: soft weighting and recycling already recover a good deal, and our method must beat them rather than a hard-crop strawman.
On the boundary side, Polyper is the method whose interface we replace. It erodes and dilates the coarse mask with fixed kernels to form "interior" and "potential boundary" regions, then uses cross-attention from interior features to refine the edge. The authors acknowledge that the boundary width is fixed. MISNet and PraNet-style reverse attention attack the same problem by erasing confident foreground so later layers focus on edges. Wang et al.'s left atrium losses push predictions away from 0.5 at edges. Spectral U-Net is "spectral" in the wavelet sense only. It keeps edge detail through downsampling, but it still ends in a plain argmax, so it is a complementary backbone choice rather than a competitor.
Core Claim
Let $X\in\mathbb{R}^{n\times k}$ be the top $k$ eigenvectors of a normalized affinity graph over $n$ superpixels in an enlarged window, or in the whole image at reduced resolution. Affinities come from decoder features of the first-stage network plus position. Let $A$ be the set of nodes the first stage labels confidently, split into lesion and background. Let $U$ be all remaining nodes. We solve
$$\min_{R,\;Y_U}\ \|XR-Y\|_F^2 \quad \text{s.t. } R^\top R=I_k,\ Y_A \text{ fixed},\ \text{each row of } Y_U \text{ one-hot},$$
where $R$ is a $k\times k$ rotation and $Y\in\{0,1\}^{n\times k}$ stacks the fixed anchor rows $Y_A$ and the free rows $Y_U$. The lesion and background anchors occupy two of the $k$ columns. The other $k-2$ columns are unanchored segments. The alternating updates are unchanged from Yu and Shi: the rotation step is an SVD of $X^\top Y$, and the assignment step is a row-wise argmax over $U$ only.
We use the per-node misfit $r_i=\|(XR)_i-Y_i\|_2$ in two ways. It defines an adaptive boundary band, the nodes with high $r_i$, which replaces Polyper's morphological band as input to the boundary-sensitive attention. It also serves as an uncertainty map. An unanchored segment that is compact and has a weakly elevated mean first-stage probability is promoted to a candidate lesion and passed to the refinement stage. This is the mechanism by which recall can exceed first-stage recall.
Research Program
The first phase is a plug-in study. We keep a trained Polyper-style two-stage network fixed and swap only the interface between stages. That interface is the band extraction and, optionally, candidate promotion. This isolates whether the anchored rotation helps at all, with no extra training and a cost dominated by one small eigendecomposition per image. Yu and Shi report 0.1 to 1.1 seconds for discretizing an 11,000-pixel image, which suggests superpixel graphs are cheap.
The second phase trains the refinement stage on the new bands. Gradients flow through the $k\times k$ SVD, which is well conditioned. Gradients through the graph eigenvectors are stopped at first because near-equal eigenvalues make them unstable. We will then test whether unfreezing them through a smoothed eigendecomposition improves the affinities.
The third phase studies sensitivity to $k$. Yu and Shi flag the choice of cluster count as unsolved, and here $k$ controls how many unanchored segments can surface as missed candidates. We sweep $k$ from 3 to 8 and report the full curve rather than a tuned optimum.
Baselines
The do-nothing reference is the first-stage prediction thresholded at 0.5, with no refinement. Every gain is reported relative to it, so readers can see how much any refinement adds.
The strongest existing method on the polyp benchmarks is Polyper with its fixed-width morphological boundary extraction. The file reports an mDice of 86.51% on ETIS for Polyper, against 81.00% for SegT. MISNet reports mDice of 0.918 on CVC-ClinicDB and 0.903 on Kvasir-SEG (MISNet). The file gives no Polyper numbers for Kvasir-SEG or CVC-ClinicDB, so there is no reported result on Lacuna for Polyper on those sets, and we will rerun it. We follow Polyper's training and cross-dataset protocol and use the same backbone and training images. Its kernel widths get the same validation tuning budget as our threshold on $r_i$ and our $k$. Beating it means a higher mDice on ETIS and CVC-ColonDB, the unseen low-contrast sets, over three seeds. We also require a gain in boundary-focused metrics (boundary F-score and 95th-percentile Hausdorff distance) that is larger than the gain in mDice. A mere mDice tie with better boundaries would still be informative, but it would not count as a clear win.
On pancreas CT, the strongest cascade competitor is RSTN, which reports a Dice-Sørensen coefficient of 84.50% on NIH Pancreas. We rerun it with our preprocessing and replace its saliency multiplication with our anchored interface on the same backbone. On the left atrium, the reference is Wang et al.'s full framework at an average Dice of 92.24 on the 20 test volumes (Wang et al.). Its thin pulmonary veins stress fixed-width bands most directly.
Three ablations isolate the key ingredient, all on the same graph and anchors. The first uses the entropy of the first-stage softmax as the band, which tests whether the misfit carries information beyond network uncertainty. The second runs K-means on the same eigenvectors followed by majority-vote labeling, the comparison Huang et al. made in the unsupervised setting. The third runs unanchored spectral rotation followed by post-hoc matching of segments to the coarse mask, which tests whether anchoring inside the optimization matters.
Why not just…?
Why not just propagate labels from the confident seeds over the same graph, random-walker style? This is the most serious threat. For the two-class refinement of an already detected lesion, harmonic label propagation from lesion and background seeds may place the boundary as well as anchored rotation does. We run it on the identical graph with identical seeds and tuning budget. If it matches us on detected lesions, we will say so and narrow the claim to the two things propagation cannot supply. First, it forces every node into a seeded class, so a missed lesion with no seed is absorbed into background and cannot appear as its own segment. Second, it provides no alignment misfit. The decisive comparison is therefore the missed-lesion subset and the band quality analysis below, not overall mDice.
Why not just train with boundary-aware losses end-to-end? Wang et al. show that overlap and focal positive losses lift a cross-entropy U-Net from 82.01% to 91.54% Dice on the atrial data, so loss design is powerful. These losses sharpen confident edges, however, and do nothing for lesions the first stage never activates on. We add an overlap-loss-trained first stage as a baseline and also combine it with our interface. If the two gains add up, the mechanisms are complementary. If the loss alone closes the gap on both boundary metrics and missed-lesion recovery, our interface is unnecessary.
Why not just keep soft recurrent weighting, as in RSTN or RecycleNet? Soft weighting removes the hard crop but still multiplies features by the first-stage probability. Near-zero probability over a missed lesion therefore still suppresses it, only less completely. Recurrence also multiplies inference cost. We match compute by giving RSTN its published number of passes and reporting wall-clock time for both. Only the missed-lesion analysis separates the two approaches on the recall question.
Proposed Experiments
We evaluate on ETIS-LaribPolypDB, CVC-ColonDB, Kvasir-SEG, CVC-ClinicDB, and CVC-300 for polyps, NIH Pancreas for small low-contrast organs, and the 2018 Atrial Segmentation Challenge for thin structures. Beyond mDice, mIoU, and boundary metrics, we stratify every result by local contrast across the ground-truth edge, since the claim is specifically about low-contrast margins.
Three checks decide the proposal. The first is band efficiency: for each band method (fixed-width morphology, softmax entropy, misfit $r_i$), we plot the fraction of final pixel errors that fall inside the band against band area, separately for compact and for thin or irregular targets. The misfit band should dominate on irregular targets. If it does not, the core mechanism fails. The second is missed-lesion recovery: among ground-truth lesions whose first-stage IoU is below 0.1, we measure the fraction reaching IoU above 0.5 after refinement, alongside the false-candidate rate per image. Any recovery above zero is something neither Polyper nor label propagation can produce by construction, and the cost in false candidates determines whether it is clinically useful. The third is the label-propagation parity test described above. It determines whether the paper's contribution is a better boundary refiner or, more narrowly, an uncertainty-and-recall interface.
References
- Multiclass Spectral Clustering
- Spectral Rotation versus K-Means in Spectral Clustering
- A Unified Framework for Discrete Spectral Clustering
- Unified Spectral Clustering With Optimal Graph
- Recurrent Saliency Transformation Network: Incorporating Multi-Stage Visual Cues for Small Organ Segmentation
- Parallel Detection-and-Segmentation Learning for Weakly Supervised Instance Segmentation
- Abstract: RecycleNet - Latent Feature Recycling Leads to Iterative Decision Refinement
- Polyper: Boundary Sensitive Polyp Segmentation
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- Parallel Res2Net-based Network with ReverseAttention for Polyp Segmentation
- Combating Uncertainty with Novel Losses for Automatic Left Atrium Segmentation
- Spectral U-Net: Enhancing Medical Image Segmentation via Spectral Decomposition