Segmentation models for low-contrast images, such as polyps that look like the surrounding colon wall or stroke lesions that barely differ from healthy brain tissue, are usually trained with two losses added together. A region loss (Dice or cross-entropy) rewards getting the overall area right. A boundary loss rewards putting the contour in the right place. The weight between them is hand-tuned and often ramped up over training, and it has to be retuned for every dataset. We propose to find out where in the image and under what supervision these two losses actually pull the model in opposite directions. We then resolve that disagreement pixel by pixel instead of with one global weight.
Our working hypothesis is deliberately narrow. With clean, fully annotated masks, region and boundary gradients mostly agree in sign at each pixel, and the weight sensitivity is mainly a problem of mismatched gradient size. That case is cheap to fix. The two losses start to genuinely disagree when the labels are noisy. This happens when pseudo-labels are produced by spreading a few annotations over a similarity graph built from imperfect backbone features. The disagreement then concentrates in the thin, ambiguous band around low-contrast edges. If this holds, the right balancing tool is a per-pixel one, and it matters most in weakly supervised medical segmentation.
Core Claim
Label-efficient pipelines build pseudo-masks by propagating sparse annotations over a graph built from backbone features. If those starting features are noisy, the graph gains edges that cross object boundaries and the propagated labels spill across the true edge. Two lines of work show this cascade in general graph learning. Lu et al. show that poor starting embeddings cause propagation to compound errors. Huang et al. show that a graph derived from features cannot recover structure the features lack. In our setting the spill has a specific consequence. The region loss is computed on soft propagated probabilities that hover near 0.5 in the ambiguous band. The boundary loss, however, needs a distance map computed from a thresholded hard mask, and that map asserts a crisp inside/outside sign in exactly the same band. The two targets disagree locally.
A single global weight, whether a schedule or a whole-image Pareto solver, must trade this local disagreement off against thousands of pixels where the two losses agree. We claim that solving a two-gradient balancing problem independently at each pixel does three things. It reduces sensitivity to the loss weight. It improves boundary accuracy under noisy pseudo-labels. It gives little or no gain under clean labels, where simple gradient-size normalization should suffice. That last prediction is a falsifiable part of the claim, not a hedge.
Mechanism
The method reuses the key trick of Multi-Task Learning as Multi-Objective Optimization. That paper computes a common descent direction with the Frank-Wolfe min-norm solver at a shared representation, so only one backward pass is needed (MGDA-UB). It treats a shared encoder output as that representation and solves for one weight per task per batch.
We instead take the output logit map as the shared representation. Both region and boundary losses act on the same logits, so the output-layer bottleneck that MGDA-UB assumes exists trivially. This also sidesteps the known limitation that MGDA-UB handles losses injected at multiple depths poorly. We then solve the two-task min-norm problem separately at every pixel.
Let $g_r(i)$ and $g_b(i)$ be the gradients of the region and boundary losses with respect to the $C$ class logits at pixel $i$. The min-norm convex combination has a closed form:
$$a_i = \mathrm{clip}\!\left(\frac{(g_b(i)-g_r(i))^\top g_b(i)}{\lVert g_b(i)-g_r(i)\rVert^2},\,0,\,1\right), \qquad g(i) = a_i\,g_r(i) + (1-a_i)\,g_b(i).$$
Here $a_i$ is the weight on the region gradient at pixel $i$. The combined map $g$ is backpropagated through the network once. The added cost is a few per-pixel vector operations.
Two properties matter. First, where the two gradients agree, the solution collapses toward the smaller one. We therefore also test a variant that normalizes each gradient to unit length before solving, so that gradient size and gradient direction are handled separately. Second, where they oppose, $g(i)$ shrinks toward zero. The model effectively stops committing at pixels whose targets contradict each other. This is desirable if the contradiction reflects label noise. It is harmful if it stalls learning of true edges, and our experiments must measure which effect wins.
Per-pixel stationarity in logit space does not imply Pareto stationarity in parameter space. We state that openly and measure parameter-space conflict separately.
The boundary loss is the distance-map loss of Boundary loss for highly unbalanced segmentation. It multiplies predicted probabilities by a precomputed signed distance to the target contour, and that paper pairs it with Generalized Dice under a hand-set ramp. We also run the prediction-driven Active Boundary Loss for Semantic Segmentation. That loss moves predicted boundaries along distance-transform directions. Its authors already observed local "tug-of-war" gradients between neighbouring pixels and handled them with an ad hoc stop-gradient rule. Our per-pixel solver is a principled replacement for that rule, and its gradients are more likely to conflict even under clean labels.
Research Program
The first stage is diagnostic and could stand alone as a result. For fixed networks at several points in training, we compute per-pixel cosine similarity between $g_r$ and $g_b$ in logit space. We also compute cosine similarity between the two losses' full parameter gradients. We bin pixels by distance to the true contour and by local image contrast.
We run this under a ladder of supervision noise. The ladder starts from clean full masks. It moves to scribble and box annotations simulated from ground truth and expanded into pseudo-masks. The expansion uses the tree-based affinity propagation of Label-efficient Segmentation via Affinity Propagation, which builds a minimum spanning tree over backbone features and spreads labels along it. At the top of the ladder, the same propagation runs on deliberately degraded features: Gaussian noise injected into the backbone features before the tree is built, and a weaker backbone. This gives a controlled knob for the starting feature noise studied in Direction 1.
The hypothesis predicts three things. Sign conflict should be rare under clean masks. It should rise with feature noise. It should concentrate within a few pixels of low-contrast contours. If conflict is instead spread uniformly or absent under noise, the per-pixel method has no reason to exist, and we would report the diagnostic as the contribution.
The second stage evaluates the balancing rule across the same ladder. We report accuracy and also a sensitivity measure: the spread of final Dice across a seven-point grid of the nominal boundary weight. The claim is about removing retuning, so a method that wins at its best weight but is equally fragile has not delivered.
We deliberately do not add a graph-smoothness loss as a third objective. The Dirichlet-energy regularizer of Joint Feature and Differentiable $ k $-NN Graph Learning using Dirichlet Energy smooths features over a learned graph. It was validated only on tabular and synthetic data, and it would smooth straight across the faint edges we care about. In this proposal the graph enters only as the pseudo-label source. A three-objective version is a follow-up once the two-objective behaviour is understood.
Baselines
The naive reference is Generalized Dice or Dice plus cross-entropy alone, the compound default recommended by Segmentation Loss Odyssey. Under clean labels, Boundary loss for highly unbalanced segmentation reports Dice 0.575 for GDL alone on ISLES and 0.727 on WMH.
The strongest existing loss-balancing competitor is that paper's own GDL plus boundary loss with its hand-tuned rising weight. It reports Dice 0.656 on ISLES, and Dice 0.748 with a 10% lower Hausdorff distance on WMH. It is the strongest because the schedule encodes real knowledge: region loss first for localization, boundary loss later for refinement. We rerun it ourselves with our backbone and data splits and give its schedule the same number of tuning trials as our method's single temperature-free rule receives in total (none beyond the nominal weight grid). Beating it means matching or exceeding its best-tuned Dice and 95th-percentile Hausdorff distance under clean labels, and exceeding it under noisy pseudo-labels, with a markedly smaller spread across the weight grid.
The strongest global gradient-balancing competitor is MGDA-UB from Multi-Task Learning as Multi-Objective Optimization, applied with one weight per image. It is joined by Conflict-Averse Gradient Descent for Multi-task learning, which searches for an update that improves the worst-off loss within a neighbourhood of the average gradient. MGDA-UB's reported gain is on CelebA: 8.25% average error versus 9.62% for uniform weighting. No reported result was found on Lacuna for either method balancing region and boundary segmentation losses, so we run both on identical data and compute.
For low-contrast polyp segmentation, the architectural reference is Multi-scale Information Sharing and Selection Network with Boundary Attention for Polyp Segmentation. It reports mDice 0.903 on Kvasir-SEG and 0.918 on CVC-ClinicDB with fixed loss weights. Our rule can be applied on top of it, so the fair comparison is MISNet with its losses versus MISNet with the same losses balanced per pixel.
The isolating ablations follow from the mechanism. We replace the per-pixel weights with a fixed band mask around the pseudo-contour whose average weight matches the per-pixel solver. We replace the min-norm rule with per-pixel gradient-size normalization only. We replace it with a per-pixel projection that removes the conflicting component without shrinking.
Why not just…?
Why not just tune the warm-up schedule carefully? This is the most credible threat under clean labels, and we expect it to roughly tie our method there. Generating Images with Perceptual Similarity Metrics based on Deep Networks and Domain Separation Networks document that such weights are dataset-specific and fragile. The more specific problem is that a schedule is one number per epoch for every pixel and every image. It cannot tell an ambiguous boundary band with contradictory pseudo-labels from a clean interior. The weight-grid spread under the noise ladder settles this. If the tuned schedule's spread and accuracy match ours at high feature noise, the per-pixel idea fails.
Why not just use MGDA or conflict-averse descent globally? These methods resolve conflict between loss vectors summed over the whole image. If disagreement is confined to a thin band, summing it with agreeing interior pixels dilutes it, and a global solver sees little conflict. The diagnostic stage directly measures how much image-level cosine similarity hides the pixel-level distribution. Head-to-head runs with identical backward-pass budgets then test whether that hidden conflict matters for accuracy.
Why not just upweight a fixed band around the contour? This is a serious alternative. If our hypothesis about where conflict lives is right, a static band weight might capture most of the gain without any solver. The matched-weight band ablation exists for this reason. Per-pixel balancing earns its place only if it beats the band mask under noisy pseudo-labels. That would show the conflict depends on the local pair of targets, not just on distance to the edge. If the band mask ties, we report that and recommend it.
Why not just add a boundary branch to the architecture? Branch designs such as MISNet and DB-MSMUNet:Dual Branch Multi-scale Mamba UNet for Pancreatic CT Scans Segmentation improve the forward pass but still sum fixed-weight losses into a shared encoder. The DB-MSMUNet authors also note the added parameter cost. Our rule is orthogonal to these designs, and the MISNet-plus-balancing comparison tests whether it adds anything on top of the best boundary architecture in the file.
Proposed Experiments
The medical benchmarks are ISLES and WMH, where published boundary-loss numbers exist. We add Kvasir-SEG, CVC-ClinicDB and ETIS-LaribPolypDB for low-contrast polyps; ETIS is the hardest, with flat and hidden polyps. We add the NIH Pancreas CT set for low soft-tissue contrast, where DB-MSMUNet reports 89.47% Dice. Each is run under the four-rung supervision ladder: full masks, simulated scribbles, simulated boxes, and boxes with feature-degraded propagation. All methods share the backbone, augmentation, epochs, and per-method tuning budget. We report means over three seeds.
Metrics are Dice, 95th-percentile Hausdorff distance, boundary F-score within a two-pixel tolerance, and the weight-grid spread of Dice. We also report per-pixel and parameter-space conflict statistics and wall-clock overhead.
The decisive checks follow directly from the claim. Conflict must increase with injected feature noise and localize near low-contrast contours. The per-pixel rule must beat the tuned schedule, global MGDA-UB, and the matched band mask at the upper rungs of the ladder, while roughly tying at the clean rung. The accuracy-versus-noise curve should widen the gap as noise grows. If per-pixel gains appear equally at the clean rung, the mechanism is size normalization rather than conflict resolution, and the gradient-size-only ablation will show it. If they appear at no rung, the hypothesis is wrong and the diagnostic study is the result.
References
- Unified Constraint Propagation on Multi-View Data
- Beyond Graph Convolution: Multimodal Recommendation with Topology-aware MLPs
- Multi-Task Learning as Multi-Objective Optimization
- Boundary loss for highly unbalanced segmentation
- Active Boundary Loss for Semantic Segmentation
- Label-efficient Segmentation via Affinity Propagation
- Joint Feature and Differentiable $ k $-NN Graph Learning using Dirichlet Energy
- Segmentation Loss Odyssey
- Conflict-Averse Gradient Descent for Multi-task learning
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- Generating Images with Perceptual Similarity Metrics based on Deep Networks
- Domain Separation Networks
- DB-MSMUNet: Dual Branch Multi-Scale Mamba UNet for Pancreatic CT Scans Segmentation