Medical image segmentation models often fail at one specific place: the faint border between a target and tissue that looks almost the same, such as the right ventricle against myocardium on cardiac MRI, the prostate capsule against surrounding tissue, or a polyp against colon wall. Labels are scarce in these domains, so the usual remedy is to pre-train the network on unlabeled scans from the same domain first. Most current pre-training objectives, however, either ignore boundaries entirely or define "these pixels belong together" using a flat over-segmentation (superpixels, meaning small compact clusters of similar pixels) or raw intensity gradients. In exactly the low-contrast regions that matter, both of these are unreliable.
We propose to pre-train dense encoders so that their feature space respects a region hierarchy: a tree in which pixels merge into regions, regions into larger regions, and so on up to the whole image. The key move is that this tree is not fixed in advance from image intensities. It is recomputed during training from a slowly updated copy of the network's own features. The training signal is a ranking constraint: if two pixels merge earlier in the tree than a third, their features must be closer. This removes the need to pick a single superpixel scale. Because the hierarchy is built from learned features rather than gradients, it can in principle come to separate tissues that intensity edges cannot. The core claim is testable: such pre-training should improve few-label segmentation, and its gains should be concentrated on the lowest-contrast stretches of ground-truth boundary.
Core Claim
A hierarchical segmentation can be written as an ultrametric. This is a distance $u(p,q)$ between pixels $p$ and $q$, defined as the "altitude" at which they first fall into the same region. It satisfies $u(p,r) \le \max(u(p,q), u(q,r))$, so once two regions merge they stay merged at every coarser scale. Najman's work established that hierarchies and ultrametric watersheds are in one-to-one correspondence (Najman). The Binary Partition Tree by Altitude Ordering builds such a tree from any edge-weighted graph using Kruskal's minimum-spanning-tree algorithm (Cousty and Najman).
We use the ultrametric of a hierarchy computed on teacher features as a graded, scale-free supervisory signal for a student encoder. For sampled pixel triplets $(p,q,r)$ with $u(p,q) < u(p,r)$, the student's feature distance $d$ is trained with a margin ranking loss:
$$\mathcal{L}_{\text{ultra}} = \mathbb{E}_{(p,q,r):\,u(p,q)<u(p,r)} \big[\max(0,\ d(p,q) - d(p,r) + m)\big],$$
Here $m$ is a margin. Triplets are sampled preferentially where $q$ and $r$ lie on opposite sides of edges with low merge altitude, meaning faint, easily-crossed borders.
Only the order of merges matters, not their absolute altitudes. As a result, there is no superpixel count, no compactness parameter and no single partition scale to tune. The teacher is an exponential moving average of the student. Early in training its hierarchy comes mostly from intensities. As the encoder improves, the hierarchy is increasingly built from learned features, and it can begin to cut along boundaries that intensity gradients never expose.
How This Differs From Existing Work
The closest work motivates the proposal by exposing the gap. SuperCL builds contrastive positive pairs from flat SLIC superpixels, aligning pixels within a superpixel and averaging superpixel features across images, and reports average gains of 3.98% DSC over the previous best baseline at 10% labels across eight medical benchmarks (Zeng et al.). It commits to one scale chosen by hand, and its authors acknowledge sensitivity to the superpixel generator. Averaging over superpixel bodies also dilutes exactly the border pixels where ambiguity lives.
Boundary-aware Information Maximization pre-trains with a global contrastive term, a clustering term over a fixed number of clusters, and a loss aligning cluster entropy with intensity gradients (Boundary-aware information maximization for self-supervised medical image segmentation). Its boundary signal is therefore tied to image gradients and a flat cluster count. We replace both with a learned hierarchy.
LoCo shows that focusing contrast on ambiguous edge pixels helps, but only during semi-supervised fine-tuning with class prototypes (Cai et al.). We move a similar emphasis upstream, without labels. The emphasis is defined by low merge altitude rather than prediction-confidence thresholds.
Classical hierarchy methods supply the machinery but have stayed offline. Structural-entropy hierarchical superpixels run fast on BSDS500 but are hand-tuned and untested on medical data (Peng et al.). Tree-of-Shapes features gave contrast-invariant damage detection on satellite imagery (Gueguen and Hamid). Neither has been used as a learning target for a deep network.
PrimGeoSeg constrains our expectations in a useful way. Pre-training on synthetic 3D primitives with sharp artificial boundaries reaches 82.0 Dice on BTCV (Tadokoro et al.), which shows that geometric boundary priors transfer even without real data. Our bet is that real, fuzzy, nested tissue structure provides a prior those clean shapes cannot.
Research Program
The method has three parts.
The first part builds the hierarchy. At each step the teacher encoder produces a feature map at reduced resolution, for example one-quarter. We form a 4-connected grid graph whose edge weights are cosine dissimilarities between neighboring features, mixed with intensity dissimilarity through a weight $\alpha$ that decays from 1 to a small floor over training. We then run Kruskal's algorithm to obtain the binary partition tree. On a grid of a few thousand nodes this costs about the same as one data-augmentation step, and it can run on CPU workers ahead of the GPU.
The second part samples triplets and computes the loss. For each anchor we draw a near partner and a far partner using lowest-common-ancestor altitudes. Sampling is biased so that partners straddle the lowest-altitude 20% of tree edges. We combine $\mathcal{L}_{\text{ultra}}$ with a standard global contrastive or self-distillation term, because a purely self-referential tree can collapse. Two guards address collapse: the intensity floor, and normalizing altitudes by rank. Collapse would show up as tree depth shrinking or all merges bunching at one altitude, and we will monitor both throughout training.
The third part is analysis without labels. We take the teacher hierarchy at the end of pre-training and measure how well its best single cut covers each ground-truth organ mask. We compare this against SLIC and against intensity-only trees. This tells us whether the learned tree actually discovers anatomy at low-contrast borders, separately from whether downstream Dice improves.
All pre-training is 2D and slice-based on the same unlabeled pools used by SuperCL and Boundary-aware Information Maximization. This keeps the comparison controlled. A 3D variant on BTCV is a secondary goal, gated on the 2D results.
Baselines
The naive reference points are random initialization and ImageNet initialization. Both are fine-tuned with the same 2D U-Net and the same labeled subsets. They establish how much any in-domain pre-training buys.
The strongest existing method on this exact question is SuperCL (Zeng et al.). It is the published pre-training method that already uses region structure, it is evaluated on ACDC and ISIC 2018 at 10% and 25% labels, and it reports 3.98% DSC average gain at 10% labels over the prior best. The file gives no absolute per-dataset Dice for it, so we will rerun it from its released setup on our splits. Boundary-aware Information Maximization (Boundary-aware information maximization for self-supervised medical image segmentation) is the strongest boundary-focused pre-training competitor. It reports up to 17.35% average DSC improvement over its baseline on ACDC with few labeled scans, and up to 14.35% (RV) and 10.12% (Myo) when used to initialize a Mean Teacher. For PROMISE12 no reported numeric result was found on Lacuna, so we rerun it as well.
Fairness is enforced in four ways. Every method gets the same backbone, the same unlabeled pool, and the same number of pre-training epochs and GPU-hours, counting tree construction time against our method. Every method also receives the same small hyperparameter search (eight configurations each, including SLIC scale for SuperCL and cluster count for the information-maximization method). Fine-tuning is identical across methods.
Beating the strongest baseline means a mean Dice gain on ACDC and PROMISE12 at the lowest label budgets (one or a few labeled scans, and 10%) that exceeds the spread across five random seeds. It also requires a larger gain in the low-contrast boundary bins than in high-contrast bins.
Two ablations isolate the key ingredients. A fixed intensity-only hierarchy (with $\alpha$ held at 1) tests whether bootstrapping the tree from learned features is what matters. A multi-scale SLIC variant of SuperCL, with positives at three SLIC scales combined, tests whether "hierarchy" means anything beyond "several flat scales". A third ablation drops the low-altitude triplet bias to test whether the boundary focus is doing work.
For the BTCV secondary experiment, the reference is PrimGeoSeg at 82.0 Dice (Tadokoro et al.), compared under its reported fine-tuning protocol.
Why not just…?
Why not just use ImageNet weights or a medical foundation model? Prior surveys argue that natural-image features favor high-contrast silhouettes and miss subtle clinical variation (Qiu et al.; Tajbakhsh et al.). CTO reports outperforming MedSAM on abdominal CT and skin lesions with explicit boundary operators (Lin et al.). Our ImageNet baseline measures this directly, and the contrast-stratified boundary metric shows whether any gap sits where we predict.
Why not just run SuperCL with several superpixel scales? This is the most serious threat. If multi-scale flat superpixels match our method, the hierarchy itself adds nothing and the honest conclusion is "use more scales". The multi-scale SLIC ablation is built to settle this. Our prediction is that it helps on bulk Dice but not in the lowest-contrast bins, because SLIC at every scale still cuts by intensity and compactness. If that prediction fails, we will report it as the main finding.
Why not just add a boundary loss during fine-tuning? Boundary losses in supervised training help (Kang et al.; Niu et al.), but with one or a few labeled scans there are very few boundary pixels to learn from. The information-maximization results show that upstream boundary-aware pre-training adds large gains that semi-supervised fine-tuning alone does not reach (Boundary-aware information maximization for self-supervised medical image segmentation). We will also add the same boundary loss to every method's fine-tuning. This shows whether pre-training gains survive once fine-tuning is equally boundary-aware.
Why not just use a fixed, hand-built hierarchy from intensities or Sobel edges? A tree built on intensities inherits the same failure as Sobel guidance: it cannot see a border that has no gradient. Sobel edges also fire on speckle and other artifacts (Lee et al.; Farooq et al.). The fixed-tree ablation with $\alpha$ held at 1 measures exactly how much the feature-bootstrapped tree contributes.
Proposed Experiments and Decisive Checks
The primary benchmarks are ACDC cardiac MRI and PROMISE12 prostate MRI, both with severe soft-tissue ambiguity. Label budgets are one, four and eight labeled scans and 10% and 25% of the data. Secondary benchmarks are ISIC 2018 dermoscopy and Kvasir-SEG polyps at 10% labels. For context, LoCo reports 86.18% mIoU on Kvasir-SEG at 10% labels with semi-supervised training (Cai et al.). Metrics are Dice, 95th-percentile Hausdorff distance, and a boundary F-score within a 2-pixel tolerance.
The decisive analysis stratifies ground-truth boundary pixels into quintiles by the intensity contrast measured across the boundary, then reports boundary F-score per quintile. The hypothesis is supported only if our gain over SuperCL and over the fixed-tree ablation is largest in the lowest-contrast quintile.
A second check is the unsupervised covering score of the learned teacher hierarchy against ground-truth masks. If downstream Dice improves while covering does not, the hierarchy is not the reason, and the claim fails.
A third check is stability. We track tree depth and the spread of merge altitudes through training to confirm the bootstrapped hierarchy does not collapse. We also report sensitivity to the intensity floor and the triplet margin. Our claim of a scale-free method requires that performance varies less across these two settings than SuperCL's does across SLIC scales.
References
- Playing with Kruskal: algorithms for flat and hierarchical watershed cuts
- SuperCL: Superpixel Guided Contrastive Learning for Medical Image Segmentation Pre-training
- Boundary-aware information maximization for self-supervised medical image segmentation
- LoCo: Low-Contrast-Enhanced Contrastive Learning for Semi-Supervised Endoscopic Image Segmentation
- Hierarchical Superpixel Segmentation via Structural Information Theory
- Large-scale damage detection using satellite imagery
- Primitive Geometry Segment Pre-training for 3D Medical Image Segmentation
- Pre-training in Medical Data: A Survey
- Surrogate Supervision for Medical Image Analysis: Effective Deep Learning From Limited Quantities of Labeled Data
- Rethinking boundary detection in deep learning-based medical image segmentation
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- A Medical Image Segmentation Method based on Multi-scale Features and Contour Loss Constrain (S)
- Structure Boundary Preserving Segmentation for Medical Image With Ambiguous Boundary
- LSSF-Net: Lightweight segmentation with self-awareness, spatial attention, and focal modulation