Making Boundary Decisions Ignore Nuisance Feature Directions

Segmentation networks fail where targets fade into their surroundings. Examples are flat polyps blending into the colon wall and thin atrial walls in contrast-enhanced MRI. They fail worst when moved to a new hospital. This note proposes a specific explanation and a fix. The explanation is that a network's deep features are highly redundant. Their largest directions of variation encode illumination, mucosal texture and scanner character rather than anatomy. Pixels on an ambiguous boundary carry no strong evidence of their own, so the network breaks ties using exactly these nuisance directions, and these are the directions that change between sites. The fix is to require, as an explicit training constraint, that predictions in the boundary band barely respond to movement along the dominant directions of background-feature variation.

The constraint can be enforced exactly, rather than as a hand-weighted loss term, for a reason specific to this setting. Writing boundary requirements pixel by pixel produces tens of thousands of constraints per image. That is more than current constrained solvers can track. Because the features are redundant, the nuisance directions span a subspace of only a few dozen dimensions. One constraint per direction per decoder scale therefore yields a problem with fewer than a hundred constraints. Ordinary primal-dual training handles that size without difficulty. Redundancy causes the failure, and the same redundancy keeps the remedy cheap to enforce.

Core Claim

The claim has two parts. First, the generalization drop of boundary-focused polyp segmenters on unseen datasets is concentrated in the boundary band. It is driven by sensitivity to a low-dimensional subspace of high-variance background feature directions, and this subspace can be estimated without labels. Second, constraining that sensitivity with per-direction Lagrange multipliers recovers out-of-distribution boundary accuracy with little in-distribution cost. The multipliers are weights adjusted during training, one per constraint, so that each limit is met. The proposal claims this recovery happens without the tuning burden of fixed-weight penalties. It also claims the recovery is not achieved by simply deleting those directions from the features.

The first part rests on three findings. Eliminating Feature Ambiguity for Few-Shot Segmentation shows that large receptive fields leak background content into boundary features. Its AENet removes that leak with a heuristic subtraction-and-reweighting plug-in for few-shot matching. Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection shows on MVTec-AD that the highest-variance principal components of pretrained features capture background texture and lighting. Its useful signal sat in low- or intermediate-variance components scattered across the spectrum. Third, the boundary networks themselves degrade sharply off-distribution. PraNet reaches 0.898 mean Dice on Kvasir-SEG but 0.628 on ETIS. MISNet still loses 16.8% on ETIS.

None of these works constrains how a boundary decision depends on particular feature directions. AENet edits features heuristically. The eigencomponent paper selects components with labelled validation AUROC for anomaly scoring, not segmentation. The polyp networks shape boundaries through architecture and auxiliary losses.

Research Program

The method has three parts. The first is a running estimate of the nuisance subspace at each decoder scale. A frozen copy of the current network marks pixels it labels as background with high confidence and away from any predicted contour. At each scale we keep an exponential moving average of the covariance of their features across images. We take the top $k$ eigenvectors, with $k$ around 16. These are computed from the network's own predictions and need no extra labels. They can also be computed on unlabeled target-site images when such images are available.

The second part is the sensitivity constraint. Let $z(p)$ be the boundary-refinement logit at pixel $p$. Let $B$ be the uncertain band, obtained by thresholding the coarse prediction's probability between, say, 0.2 and 0.8. This band adapts to lesion size, unlike the fixed erosion and dilation kernels of Polyper. Let $u_{j,s}$ be the $j$-th nuisance direction at scale $s$, and let $F_s$ be the feature map at that scale. We require

$$c_{j,s} = \mathbb{E}_{p \in B}\big[(u_{j,s}^\top \nabla_{F_s} z(p))^2\big] \le \epsilon_{j,s}.$$

This states that, on average over the band, a small push of the features along a nuisance direction must move the logit by at most a set amount. The left side is computed with Jacobian-vector products on a random subset of band pixels. Each tolerance $\epsilon_{j,s}$ is set to a fixed fraction of the sensitivity measured along a matched direction drawn from the polyp-versus-mucosa discriminative subspace. This keeps the constraint scale-free.

The third part is the optimizer. Training minimizes the usual weighted BCE plus IoU loss of the host network. Each constraint gets its own multiplier, updated by projected dual ascent. Batch estimates of $c_{j,s}$ are noisy, so we borrow the bias-correction idea from CGM-relax in Calibrating Generative Models to Distributional Constraints. That paper estimates squared expectation penalties from independent half-batches so that small batches still give unbiased gradients. We apply the same correction to the constraint residuals.

The design deliberately avoids two failure regimes reported elsewhere. One is dual-variable explosion when constraints scale with data or pixels, flagged in Implicit rate-constrained optimization of non-decomposable objectives. The other is the $O(m^{3.5})$ cutting-plane cost that Fast Projection Onto Convex Smooth Constraints incurs once the constraint count $m$ reaches thousands. That paper is fast precisely when $m$ is small, and our formulation keeps $m$ small. It also requires convex smooth constraints, which a deep network does not give, so we take only its lesson about small $m$, not its solver. The pixel-level velocity projections of Accelerated first-order optimization under nonlinear constraints act on raw pixel variables in reconstruction problems. They are not a fit for constraints on a network's sensitivity.

As a secondary analysis, we track pairwise cosine similarity between constraint gradients during training. This is the harmonious-versus-conflicting taxonomy of Investigating Constraint Relationship in Evolutionary Many-Constraint Optimization. It tells us whether constraints at different scales are redundant and could be merged. It also tells us whether any nuisance constraint conflicts with the task loss, which would mean the direction is not truly a nuisance.

The constraint is a plug-in, so we attach it to PraNet and to Polyper without changing their architectures. The secondary regime is the Atrial Segmentation Challenge 2018 volumes. Combating Uncertainty with Novel Losses for Automatic Left Atrium Segmentation attacks the same thin, low-contrast boundaries there with Overlap and Focal Positive losses. On that dataset we attach the constraint to its 3D U-Net.

Baselines

The naive reference is each host network trained exactly as published. We use the standard polyp protocol: train on Kvasir-SEG plus CVC-ClinicDB, then test on ETIS, CVC-ColonDB and CVC-300. On Kvasir-SEG, PraNet reports mean Dice 0.898. On ETIS it reports Dice 0.628, per PraNet. We retrain rather than copy these numbers, so that seeds and preprocessing match across all arms.

The strongest existing method is Polyper. It already separates an interior region from a potential-boundary region and uses cross-attention from interior features to clean up the boundary. That is the closest architectural answer to boundary ambiguity in the file. Polyper reports 86.51% mean Dice on ETIS in Polyper: Boundary Sensitive Polyp Segmentation. That figure is far above PraNet's and CaraNet's reported 74.7%, per CaraNet, which suggests a different training split. We will rerun Polyper under our protocol before comparing.

MISNet, with reported mean Dice 0.903 on Kvasir-SEG and 0.918 on CVC-ClinicDB, is a second strong boundary-attention baseline. On the atrium data, the reference is the full OVL+FPL+recursive refinement system, reported at 92.24% Dice.

Fairness rules apply to every arm. All arms use the same backbone, the same input resolution and the same number of epochs. Each gets an equal hyperparameter search budget of the same number of trials. Our method spends its trials on $k$ and the tolerance fraction. Baselines spend theirs on learning rate and loss weights. No arm sees target-site labels.

Beating Polyper means two things together. ETIS and ColonDB mean Dice must improve by more than the seed-to-seed standard deviation across five seeds. Boundary F-score inside the uncertain band must improve by a clear margin. Both must hold while Kvasir-SEG Dice stays within one standard deviation.

Two ablations isolate the key ingredient. In the first, the same multipliers constrain sensitivity to $k$ random orthonormal directions instead of the nuisance subspace. This tests whether the choice of directions matters or whether any smoothness constraint helps. In the second, the constraint is applied to all pixels rather than only the band. This tests whether the effect is specific to the boundary.

Why not just…?

Why not just project the top principal directions out of the features? This is the most serious alternative, since it needs no optimizer at all. The greedy eigencomponent study argues against blanket removal: useful signal there was scattered unpredictably across the variance spectrum. Some high-variance background directions may also help interior classification, and removing them would cost in-distribution accuracy. Our constraint touches only boundary-band sensitivity and leaves the features intact. We run hard nullspace projection with the same $k$ as a baseline. If it matches us on both ETIS and Kvasir-SEG, the constraint machinery is unnecessary and we will report that.

Why not just add the sensitivity term as a fixed-weight penalty? A fixed weight is a Lagrangian with one frozen multiplier, and CGM-relax shows relaxed penalties can work well at scale. The argument for learned multipliers is that sensitivity differs widely across scales and directions, so a single weight under-constrains some directions and over-constrains others. We give the penalty version the same tuning budget, with a single weight and with per-scale weights. If it matches, the contribution narrows to the nuisance-subspace diagnosis itself, which would remain useful.

Why not just use color and illumination augmentation? Augmentation targets the same site-to-site shift and is cheap. It covers only the nuisance factors someone thought to simulate, whereas the subspace is estimated from the data. We include strong photometric augmentation as a baseline and combine it with our method. The constraint earns its place only if it adds a gain on top of augmentation.

Why not just use auxiliary boundary losses or post-hoc thresholding? Those are what PraNet, MISNet and the Res2Net polarization variant already do, and their off-distribution drops are what motivates this work. They also act as baselines here through the host networks.

Proposed Experiments

The decisive first check is diagnostic and needs no new training. For trained PraNet and Polyper models, we estimate the nuisance subspace on Kvasir-SEG background pixels and then measure three things. The first is how much of the shift in mean features between Kvasir-SEG and ETIS falls inside the subspace. The second is whether band-pixel sensitivity along these directions is larger than along random directions. The third is whether, image by image, that sensitivity predicts boundary errors on ETIS. If the shift does not concentrate in the subspace, or sensitivity does not predict errors, the hypothesis is wrong and we stop.

If the diagnostic passes, we run the full comparison on the polyp protocol. We report mean Dice, mean IoU and boundary F-score inside the band, across five seeds. On the atrium dataset we report Dice and wall-specific boundary error against the published loss stack.

Two further checks probe the method's assumptions. We track constraint satisfaction on held-out sites to see whether a limit met on training data still holds off-distribution. We track the gradient-cosine relationships between constraints to see whether some can be merged without loss. Finally, we re-estimate the subspace on unlabeled ETIS images to test whether target-side estimation adds a gain over source-only estimation.