Transport Losses for Fragmented Masks That Survive Diffuse Background Probability

A segmentation network that predicts a lesion in the wrong place gets no useful signal from overlap losses. Dice or IoU (intersection over union) is equally bad whether the predicted blob sits one pixel or one hundred pixels away, so its gradient carries no direction. The natural fix is to compare the predicted map and the target mask by how far probability mass would have to move to turn one into the other. That quantity is the Wasserstein, or optimal transport, distance. Box detectors already do this by treating each box as a Gaussian. We argue that the main obstacle to using spatial transport for dense masks is not compute. It is the faint probability that every softmax map spreads across the whole image. Once the map is normalized into a distribution, that diffuse background mass sits on average far from the target, can dominate the transport cost, and can point the gradient the wrong way. This is the same additive-noise sensitivity that has been characterized for Wasserstein distances on images.

We propose a mask loss with two parts. The first is a balanced transport term between the sharpened, normalized prediction and the target mask, which gives "where is the mass" gradients even from far misses and from many disconnected components. The second is a separate mass term that tells the network "how much foreground" to predict. We will analyze when sharpening restores a correct gradient direction. We will then test whether this combination trains fragmented-lesion segmenters from scratch, without the warm-up schedule that existing distance-based losses need, and whether it beats a much simpler Gaussian moment-matching loss specifically on targets with many components.

What exists and where the gap actually is

Two box-detection papers supply the core move. Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Loss maps each oriented box to a 2D Gaussian and uses the closed-form Wasserstein distance between Gaussians as the loss. A Normalized Gaussian Wasserstein Distance for Tiny Object Detection applies the same distance to tiny axis-aligned boxes, exponentially normalized. Both show that transport yields gradients under complete non-overlap. Both are confined to one unimodal Gaussian per object, so they say nothing about a mask made of twenty scattered lesions.

On the dense side, two papers motivate the gap and also constrain the design. Learning Generalized Intersection Over Union for Dense Pixelwise Prediction (PixIoU) weights false positives and false negatives by their distance to a single object center. Its authors concede that it struggles with intertwined or fragmented shapes. Boundary loss for highly unbalanced segmentation integrates the softmax output against a precomputed distance map of the ground-truth contour. That loss is one-directional and cannot start training from empty predictions, so it is scheduled together with Generalized Dice Loss.

Generalised Wasserstein Dice Score does use a Wasserstein distance in segmentation, but only over class labels. It remains blind to pixel coordinates. Our closest neighbor is therefore not a method in the file. It is the obvious "just put a Sinkhorn loss on the pixel grid" idea that the file's Gap 1 describes.

The theory paper that constrains us is Quantifying the noise sensitivity of the Wasserstein metric for images. It bounds how much additive pixel-wise noise inflates the Wasserstein distance between images treated as measures on grids. A softmax map with a small foreground probability everywhere is exactly such a noisy measure. We read that result as a prediction that naive grid transport will be dominated by background, and the proposal is built around defusing it.

Core Claim

Let $p(x)\in[0,1]$ be the predicted foreground probability at pixel $x$, $g(x)\in\{0,1\}$ the target mask, and $\gamma\ge 1$ a sharpening exponent. We define

$$\mathcal{L}=W_\varepsilon\!\left(\frac{p^\gamma}{\sum_x p^\gamma},\ \frac{g}{\sum_x g}\right)+\lambda\,\left|\log\frac{\sum_x p}{\sum_x g}\right|.$$

In this loss, $W_\varepsilon$ is the entropically smoothed transport cost with Euclidean ground cost on pixel coordinates (computed by Sinkhorn iterations on a downsampled grid), and $\lambda$ weights the log-mass mismatch. The first term moves mass and does not care how much mass there is. The second fixes the amount of foreground and does not care where it is. Separating the two avoids a known flaw of normalized transport: a tiny but well-placed prediction would otherwise score perfectly.

The sharpening exponent is the key ingredient. Suppose $N_b$ background pixels carry probability about $b$ and $N_f$ foreground pixels carry probability about $f$. The fraction of normalized mass sitting in the background is then

$$\beta(\gamma)=\frac{N_b\,b^\gamma}{N_b\,b^\gamma+N_f\,f^\gamma}.$$

The derivation is direct: it is the background share of the total mass after raising each pixel to the power $\gamma$. For a lesion covering 0.1% of an image with $b=0.02$ and $f=0.5$, $\beta(1)$ is about 0.44. Since background mass sits at roughly image-diameter distances, it can control the gradient. With $\gamma=3$, $\beta$ falls below 0.01.

The claim we test has three parts. First, a moderate $\gamma$ restores a target-pointing gradient from empty or far-miss states. Second, this works for any number of components, because transport matches mass to mass rather than center to center. Third, it does so without Dice warm-up. The cost of sharpening is that gradients concentrate on high-probability pixels, which may slow discovery when the prediction is uniformly near zero. The mass term exists partly to push the network out of that state.

Research Program

The first stage is analytic and synthetic. On 2D grids we place targets with one to fifty components and predictions offset by controlled distances, with uniform background floors $b$ ranging from 0 to 0.1. For each loss we measure the cosine between the negative gradient and the ideal displacement field, which is the field given by the exact unregularized transport plan. This produces maps of where each loss's gradient is informative, as a function of offset, component count, and $b$. We expect plain IoU to show zero signal beyond overlap. We expect PixIoU and Gaussian moment matching to fail as component count grows, and unsharpened transport to fail as $b$ grows. The prediction is that sharpened transport stays informative across the whole range.

The second stage is training on medical segmentation where targets are small, numerous, and fragmented. We use WMH, whose white-matter lesions are typically dozens of small fragments, and ISLES. BraTS'15 serves as a single-mass contrast case where we expect no advantage over moment matching.

The third stage checks transfer to natural images: Cityscapes small classes (traffic lights, signs, riders) and VOT-2020 masks.

Compute is handled by running Sinkhorn on a 64×64 or 96×96 pooled version of $p^\gamma$ and $g$, which gives a few thousand support points per image and is cheap on a GPU. A sliced Wasserstein variant serves as the cheaper ablation; the sliced construction appears in the kernel setting of Distribution Regression with Sliced Wasserstein Kernels, which we reuse only for its projection trick. Our expectation is that sliced transport blurs component-to-component matching in 2D, and we will measure that rather than assume it.

Baselines

The naive reference point is Generalized Dice Loss alone with the same network, which tells us how much any distance-aware term contributes at all. On ISLES the boundary-loss paper reports a GDL Dice of 0.575, and on WMH a GDL Dice of 0.727 (Boundary loss for highly unbalanced segmentation).

The strongest existing dense competitor on these benchmarks is GDL combined with boundary loss under its rebalancing schedule. It is the one published method in the file that adds a distance-aware gradient to exactly these unbalanced lesion tasks. Its reported Dice scores are 0.656 on ISLES and 0.748 on WMH, with a roughly 10% Hausdorff reduction on WMH (Boundary loss for highly unbalanced segmentation). We will rerun it ourselves with the same architecture, preprocessing, epoch budget, and an equal hyperparameter-search budget, about twenty trials each. That budget covers its schedule increment for boundary loss and $\gamma$, $\lambda$, $\varepsilon$, and grid size for ours. We count it as beaten only if our loss matches or exceeds its Dice while lowering the 95th-percentile Hausdorff distance and improving lesion-wise recall on WMH, averaged over at least three seeds. In addition, our loss must reach that result without any Dice warm-up. Training our loss alongside GDL is reported separately so that the comparison is not confounded.

PixIoU (Learning Generalized Intersection Over Union for Dense Pixelwise Prediction) is the strongest competitor for non-overlap gradients on masks. It reports 75.6% mIoU on VOT-2020 against 74.2% for an IoU loss, and a 0.6–0.8 mIoU gain on Cityscapes. We reimplement it on WMH and ISLES under the same budget, and on VOT-2020 we compare against its reported 75.6%.

The baseline we consider most dangerous is our own construction: a Gaussian moment-matching loss. It fits a mean and covariance to the soft predicted mask and to the target, then applies the closed-form Gaussian Wasserstein distance from Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Loss, plus the same mass term. No reported result was found on Lacuna for this loss on any segmentation benchmark.

The ablations isolate the ingredients. Setting $\gamma=1$ isolates sharpening. Removing the mass term isolates the separation of "where" from "how much." Swapping Sinkhorn for sliced transport isolates exact component matching.

Why not just…?

Why not just match the soft mask's centroid and covariance with a Gaussian Wasserstein loss? This is cheap, smooth, gives far-miss gradients, and borrows directly from the two box papers. On single-blob targets we expect it to tie with or beat our loss, and BraTS is included to show that honestly. Its failure should appear when the target has several separated components. A Gaussian fitted to twenty lesions has its mean in empty tissue, so the loss pulls mass toward a point where no lesion exists. This is the same single-center failure PixIoU reports. The decisive test is WMH results stratified by lesion count, together with the synthetic cosine maps against component count. If moment matching is not clearly worse on high-count scans, the transport machinery is not justified, and we will report that outcome as the finding.

Why not just use boundary loss with its Dice schedule, since it already works? It does work, and its WMH and ISLES numbers are the bar we must clear. Its gradient is defined only relative to the ground-truth contour, and the authors themselves state that it cannot train from empty predictions. Our from-scratch runs without warm-up test directly whether a symmetric transport term removes that dependency. If boundary loss with its schedule still wins on Dice and Hausdorff distance, the remaining contribution reduces to removing the need for a schedule, and we would say so.

Why not just put a standard Sinkhorn loss on the raw softmax map? This is the "obvious" version of the idea, and the $\gamma=1$ ablation is that method. The noise-sensitivity analysis in Quantifying the noise sensitivity of the Wasserstein metric for images predicts that diffuse background inflates and distorts the distance. The $\beta(\gamma)$ calculation suggests the effect is severe for lesions occupying under 1% of the image. If $\gamma=1$ performs as well as sharpened transport on WMH, our central mechanism is wrong.

Why not just use coordinate-style distance penalties such as DIoU on mask centers? Distance-IoU Loss shows that center penalties converge fast for boxes. A mask has no single meaningful center when it is fragmented, so this reduces to the moment-matching case above and is settled by the same stratified test.

Proposed Experiments

The first decisive check is the synthetic gradient-cosine map. Sharpened transport must keep a cosine above 0.5 with the ideal displacement at $b=0.05$ and fifty components, the regime where unsharpened transport and moment matching are expected to drop.

The second is from-scratch training on WMH and ISLES with no GDL warm-up, reporting Dice, 95th-percentile Hausdorff distance, lesion recall, and lesion F1. The target is boundary loss with its schedule, at its reported 0.748 and 0.656 Dice and in our own reruns. The WMH metrics follow the protocol of Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities.

The third is lesion-count stratification against the Gaussian moment-matching baseline, which is where the method should earn its existence.

The fourth is a training-speed comparison in epochs to a fixed Dice. The file's tension between smooth transport losses and slower convergence (A Normalized Gaussian Wasserstein Distance for Tiny Object Detection needed 24 epochs against 12 for IoU) could recur here, and we will report it rather than hide it.

Cityscapes small classes and VOT-2020 then test whether any gains carry outside medical imaging. On VOT-2020 the comparison is PixIoU's reported 75.6% mIoU.