Choosing Source Images by How Their Boundaries Fade

Segmentation models for colonoscopy, CT, and other medical images fail mostly at object edges. They fail where a polyp blends into the surrounding mucosa, or where an organ's tissue looks almost the same as its neighbor's. When such a model is moved to a new hospital or scanner, the usual fix is to pick training images from other datasets that "look like" the target. "Look like" is normally judged by global similarity: overall structure, color, or a whole-image embedding. Those are exactly the statistics that ignore edges. This note proposes choosing source training images by a different test: whether the boundaries in each source image fade the way boundaries fade in the target domain. That means comparing the contrast across the edge, how wide the blurry transition band is, and how strong the edge gradient is relative to local noise.

The bet is simple. If the target domain is dominated by faint, wide, camouflaged boundaries, a source pool full of crisp, high-contrast polyps teaches the wrong edge behavior, however similar those images are in global appearance. Selecting the subset whose boundary profiles match the target should improve accuracy at the edges, where the remaining error lives. Global similarity selection at the same data budget should not. Because target masks are unavailable during adaptation, the target's boundary profile must be estimated from an unlabeled model's own uncertain predictions. Whether that estimate is reliable enough is one of the central things this work tests.

Core Claim

For cross-dataset segmentation of low-contrast structures, a source subset chosen by matching edge-band statistics to the target will beat three alternatives on boundary-sensitive metrics: the full pooled source, a random subset of the same size, and a subset of the same size chosen by global image similarity. The gains should concentrate on the target images whose boundaries have the lowest contrast.

This differs from the closest work in three specific ways:

  • Not All Areas Are Equal learns pixel-, region-, and image-level weights on source data using an adversarial discriminator. It was tested on synthetic-to-real driving scenes with sharp object edges, and its weights track appearance and style. Our proposal uses no adversarial training, and its selection signal is defined on the boundary band itself.
  • PRIME prunes polyp datasets without training. It builds a similarity graph from global SSIM and Pearson correlation and keeps the central images of each cluster. It works within a single domain, and the authors note that aggressive pruning loses rare low-contrast cases. Our selection is guided by the target and is meant to keep precisely those cases when the target needs them.
  • Adaptive Domain Learning rejects harmful source batches by trial updates checked on labeled target pairs. It motivates the idea that some source data actively hurts, but it needs labeled target data that segmentation adaptation does not have. Our selection signal requires no target labels.

Research Program

Boundary descriptor. For each source image, the ground-truth mask defines a band of pixels straddling the boundary. Within that band we compute a short descriptor $d$ with three parts:

  • the contrast $c = |\mu_{in} - \mu_{out}| / \sigma_{band}$, where $\mu_{in}$ and $\mu_{out}$ are the mean intensities just inside and just outside the boundary and $\sigma_{band}$ is the intensity standard deviation in the band;
  • the transition width $w$, the distance over which a smoothed intensity profile taken normal to the boundary rises from 10% to 90% of its range;
  • the peak gradient magnitude along the boundary, normalized by local noise.

Each of the three is computed per color channel and summarized as a median and an interquartile range over the boundary. A variant replaces raw intensities with early-layer features from a frozen pretrained encoder, which captures texture camouflage that intensity alone misses.

Estimating the target profile without labels. We train a segmentation model on the full source pool and run it on unlabeled target images. The predicted boundary band is taken where the predicted foreground probability lies between 0.2 and 0.8, widened slightly by the predicted mask's own erosion and dilation. The same descriptor is computed inside this band. The obvious danger is that the band is wrong exactly where boundaries are faintest.

We therefore measure the estimation error directly on benchmarks where target masks exist but are withheld from adaptation. As a fallback, we use descriptors that depend only on intensity profiles normal to the predicted contour, which are less sensitive to small contour misplacement.

Selection rule. Each source image $s$ is scored by $\rho(s)$, the density of its descriptor $d_s$ under a kernel density estimate fitted to the target descriptors. We keep the top fraction $q$ of source images. This is hard selection, following Hua et al. and Zhang et al., who report that hard exclusion beats soft weighting when the mismatch is severe. A soft variant that weights each image's loss by $\rho(s)$ is run as an ablation. This tests that claim in a boundary setting rather than assuming it.

Diversity guard. Selecting by density near the target could collapse onto the most typical target boundary. To prevent this, the kept fraction is stratified across target descriptor quantiles, so that rare faint-edge modes in the target pull in their matching source images. This addresses the failure PRIME reports.

Segmentation models. The selected subset trains two fixed models: a standard boundary-aware polyp network, and Polyper, which splits each prediction into a confident interior and an uncertain boundary band and refines the band using interior features. Using two architectures shows that the effect comes from the data rather than one model's inductive bias.

Fixed-kernel question. The analysis also asks whether matching transition width $w$ between source and target removes the failure that Polyper attributes to its fixed-width erosion and dilation kernels. If the source edges have widths similar to the target's, a fixed kernel should fit both.

Baselines

Pooling. The naive reference is to train on the entire pooled source with no selection. Beyond Pooling argues that pooling heterogeneous sources yields biased models, but on polyps pooling is what strong published models effectively do. Zero-shot ETIS results are the honest bar. Polyper reports an ETIS mDice of 86.51% under its own training protocol, against 81.00% for SegT. MISNet reports only a 16.8% drop on ETIS, compared with drops above 38% for competing models. We will retrain Polyper on our exact source pool and report its pooled result alongside its published figure, since the two protocols may differ.

Random subset. A random subset with the same fraction $q$ separates "less data" from "the right data." Without it, any gain from selection could be a regularization effect of training on fewer images.

Global-similarity selection. This is the strongest existing competitor. It has two forms: PRIME-style selection with SSIM and Pearson correlation, made target-guided by scoring each source image by its similarity to target images, and nearest-neighbor selection in a frozen whole-image encoder embedding. Both keep the same fraction $q$ and use the same stratification. PRIME reports 92.11% Dice on Kvasir after removing 56.2% of the data, a 0.5% drop from full training. That is in-domain, so no reported result was found on Lacuna for global-similarity source selection on a cross-dataset polyp target. We run it ourselves.

Adversarial weighting. We include the hierarchical region weighting of Not All Areas Are Equal, reimplemented on the polyp backbone. That paper reports 68.1% mIoU on GTAV plus Cityscapes to Cityscapes, against 64.8% for an FCN with a global adversarial discriminator. It needs a few labeled target images, so it runs only in the few-label setting below.

Fairness. Every method uses the same backbone, the same augmentation, the same epochs, and the same 20-trial hyperparameter search on a source-only validation split. No method sees target labels except in the explicitly few-label setting.

Ablations isolating the key ingredient. The first ablation swaps the boundary descriptor for a global descriptor of identical dimension, computed over the whole image instead of the band. The second uses an oracle target descriptor computed from true target masks, which bounds the cost of label-free estimation. Beating global-similarity selection means a gain of at least two boundary-F-score points on the faintest third of target images, consistent across three seeds, with no loss in overall Dice.

Why not just…?

Why not just pool everything and use a stronger boundary-aware network? Architectures like Polyper and MISNet already narrow the cross-site gap substantially. Selection only matters if the source pool contains edge styles that actively mislead. Adaptive Domain Learning shows that harmful source data exists in denoising, but no comparable segmentation evidence appears in the file. The pooled baseline run with Polyper settles this directly. If pooling matches selection on the faintest-boundary stratum, the claim is false.

Why not just select by global similarity, as PRIME does? This is the most direct threat. Boundary contrast may correlate with global appearance, for example darker frames having fainter edges. If so, global selection would pick nearly the same images. We measure the overlap between the two selected subsets and the correlation between global and boundary descriptors. The two methods are then compared at equal budget. If the overlap exceeds roughly 80% and the scores match, the boundary descriptor adds nothing, and we will report that.

Why not just augment source images to have fainter boundaries? One could blur edges and reduce contrast near source masks instead of discarding data. This is a real competitor, because it matches boundary statistics without losing any images. We include it as a baseline: edge-band blur and contrast reduction tuned to reach the estimated target descriptor distribution. We also combine it with selection. If augmentation alone matches selection, the useful finding is that boundary-statistic matching matters, whatever the mechanism. Synthetic blur, however, may not reproduce real texture camouflage, and the feature-space descriptor variant is designed to expose that difference.

Why not just mask out ambiguous boundary pixels instead of choosing images? DyCON argues that discarding uncertain boundary voxels hurts irregular lesions. It reports 61.48% Dice against 53.53% for BCP on ISLES-2022 with 5% labels, using a loss scaled by uncertainty instead. Our selection keeps whole images and their full boundary bands, so it does not discard edge context. We still run a confidence-masking baseline on the pooled source to confirm it falls behind.

Proposed Experiments

Main protocol. We hold out one polyp dataset at a time as the target. ETIS is the primary target because its polyps are small, hidden in folds, and low in contrast; ColonDB is the second. The remaining datasets form the source pool: Kvasir-SEG, CVC-ClinicDB, CVC-300, EndoScene, and SUN-SEG frames. Target images are used without labels for descriptor estimation. A secondary few-label setting gives 20 labeled target images, mirroring ADL's 20-pair protocol, to compare against adversarial weighting and trial-update rejection adapted to Dice.

Metrics. We report mDice and mIoU, a boundary F-score at a 2-pixel tolerance, the 95th-percentile Hausdorff distance, and Dice restricted to the true boundary band. All are additionally stratified by tertiles of each target image's true boundary contrast.

Decisive checks. There are three:

  • Mechanism. The improvement over global-similarity selection should grow monotonically as target boundary contrast decreases. A flat improvement would suggest a generic data-quality effect rather than edge matching.
  • Estimate quality. The label-free target descriptor should correlate with the oracle descriptor, and the oracle-selected subset should bound the label-free result from above. A large gap between them means the estimation step is the bottleneck and needs fixing before anything else.
  • Fixed kernels. Matching transition width should reduce the error from Polyper's fixed kernels. We test this by sweeping its erosion and dilation width, which should matter less under width-matched selection than under pooling.

Generalization check. If the polyp results hold, we run the same pipeline on cross-dataset brain MRI, with BraTS2019 as source and ISLES-2022 as target. This tests whether the descriptor transfers to 3D soft-tissue boundaries, with the caveat that the two datasets also differ in the structure being segmented.