A common recipe for models that generalize is to steer training toward "flat" minima, meaning regions of parameter space where the loss barely changes when the weights are perturbed. The recipe has strong support in learned-optimization work, where penalizing curvature during meta-training produced optimizees that generalize much better. The recipe was developed on image classification, however, and medical segmentation has a different structure. In segmentation, a large share of the loss curvature comes from a thin band of pixels along object boundaries. In low-contrast scans, such as ultrasound tumors, pancreatic lesions, or the optic cup in fundus photographs, these are the pixels the model is least sure about. If that is true, then a global flatness penalty mostly flattens the boundary. That could buy robustness to scanner shift in object interiors while blurring exactly the edges clinicians care about. The blurring should get worse when images are downsampled, because the boundary then occupies fewer, more ambiguous pixels.
We propose to test this claim directly, measure it, and act on it. The fix we test is simple: apply flatness pressure only to the interior part of the segmentation loss and leave the boundary band unregularized. We evaluate it on multi-center benchmarks where data heterogeneity, input resolution, and low-contrast boundaries can be varied separately. The contribution is a measured trade-off plus a cheap intervention. If the effect holds, it also gives a meta-training objective that a future learned optimizer for segmentation should use instead of the global flatness criterion it currently inherits from classification.
Core Claim
For a pixel-wise cross-entropy loss, the Gauss–Newton approximation of the Hessian with respect to parameters $\theta$ decomposes over pixels:
$$ \mathrm{tr}\,H(\theta) \approx \sum_{i} p_i(1-p_i)\,\lVert \nabla_\theta z_i \rVert^2 . $$
Here $z_i$ is the logit at pixel $i$ and $p_i$ its predicted foreground probability. The weight $p_i(1-p_i)$ is the second derivative of cross-entropy in the logit. It peaks at $p_i = 0.5$, and near convergence it is close to zero for confidently classified interior pixels. Under low contrast, the pixels with $p_i$ near 0.5 lie almost entirely in a band around the true boundary. That band widens under scanner shift and under coarser input resolution. Our claim is that the measured sharpness of a medical segmentation loss is dominated by this boundary band. Global curvature penalties therefore trade boundary accuracy for interior robustness. Restricting the penalty to the interior should keep most of the cross-site Dice gain without the boundary loss.
The intervention splits the loss into an interior term $L_{\text{int}}$ and a boundary-band term $L_{\text{bd}}$. The band covers pixels within $w$ pixels of the ground-truth contour, with $w$ fixed in physical units so it means the same thing at every resolution. The objective is
$$ \min_\theta \; L_{\text{int}}(\theta) + L_{\text{bd}}(\theta) + \lambda\,\widehat{\mathrm{tr}}\,\nabla^2_\theta L_{\text{int}}(\theta), $$
where $\widehat{\mathrm{tr}}$ is a Hutchinson stochastic trace estimate and $\lambda$ sets the strength of the flatness pressure.
Why This Is Open
Three groups of prior work shape the idea.
Two papers motivate it. Learning to Generalize Provably in Learning to Optimize separates optimizer generalization from optimizee generalization. It adds Hessian-norm, Hessian-trace (Hutchinson), and local-entropy regularizers to learned-optimizer meta-training, and reports test-accuracy gains of up to 31% over vanilla learned optimizers on Conv-MNIST. That evidence comes only from classification, where no boundary band exists. We borrow its Hessian-trace estimator and question whether its global objective transfers to dense prediction. $\mu$LO shows that learned optimizers can be made to transfer across width, depth, and horizon. It also tested only classification and language modeling, so whether those update rules suit pixel-level losses is untested.
The boundary literature constrains the idea. Boundary attention heads and contour losses (Kang et al.; Niu et al.) and reverse attention (Parallel Res2Net-based Network with ReverseAttention) all treat ambiguous edges as needing more sensitivity, not less. Boundary-Aware Test-Time Adaptation aligns deep attention with shallow Sobel edge maps at inference time. That paper notes that downsampling distorts those shallow edges but does not address it. None of this work asks what the optimizer's implicit bias does to boundaries. We change the training objective and leave the architecture alone.
The heterogeneity and resolution work supplies the setting. nnU-Net and The Medical Segmentation Decathlon show that a well-configured standard pipeline is very hard to beat. They also show it collapses on low-contrast small targets, with 0.21 Dice similarity coefficient (DSC) on pancreas tumors and 0.16 on colon cancer. Gradient Alignment Improves Test-Time Adaptation (GraTa) supplies a five-domain fundus benchmark with real center shift. Efficient Segmentation: Learning Downsampling Near Semantic Boundaries shows that spending pixels near boundaries helps under a fixed compute budget, though only on sharp-edged driving scenes. We use its sampler as one resolution condition.
The obvious intersection idea is to meta-learn an optimizer for test-time boundary adaptation, combining M-L2O with BA-TTA-SAM or GraTa. We do not pursue it. M-L2O was shown only on LASSO, quadratic, and Rosenbrock objectives. GraTa already obtains an average 5.87% DSC gain over no adaptation with a closed-form heuristic. A learned test-time optimizer would itself suffer the meta-generalization gap on unseen centers. A skeptical reviewer would reasonably expect a tuned heuristic to match it. The more basic, unanswered question is what objective any optimizer, learned or hand-designed, should be pushed toward when training segmentation models.
Research Program
The work has three stages.
Stage 1: measure where the curvature is. For trained U-Nets on each benchmark, we estimate the Hessian trace and the top eigenvalues by Hutchinson and Lanczos methods. We compute these separately for $L_{\text{bd}}$ and $L_{\text{int}}$, sweeping the band width $w$. We repeat at three input resolutions (native, half, and quarter) and on in-domain versus held-out-center data. The first prediction is that the boundary band, though a small fraction of pixels, accounts for most of the trace. The second is that its share grows as resolution drops and on unseen centers.
Stage 2: measure what global flatness does. We train with no curvature penalty, with a global Hessian-trace penalty, with a boundary-only penalty, and with an interior-only penalty. Each condition is crossed with the three resolutions. We evaluate interior quality with Dice, boundary quality with the 95th-percentile Hausdorff distance and a boundary F-score at a fixed physical tolerance, and calibration within the band. If global flatness raises leave-one-center-out Dice while worsening the boundary metrics, the tension is real. It should be largest at quarter resolution.
Stage 3: test the split objective and its link to learned optimizers. If the interior-only penalty keeps the cross-center Dice gain and recovers boundary metrics, we take the boundary mask as the missing ingredient. As a follow-on contingent on Stage 3, we would replace the global flatness term in the learned-optimizer regularizer of Yang et al. with the masked term and meta-train a small optimizer on 2D segmentation optimizees. This is deliberately downstream. It is worth doing only if the hand-designed version shows the effect.
Baselines
The naive reference is a standard U-Net trained with Adam and Dice plus cross-entropy, under nnU-Net's automatic preprocessing and augmentation. It uses no curvature penalty and no adaptation, at each resolution. It is the strongest realistic competitor because nnU-Net won both Decathlon phases and reports 88.1% average Dice on the Multi-Atlas Labeling Beyond the Cranial Vault abdominal benchmark. On the hard Decathlon targets the reported numbers are 0.21 DSC for pancreas tumors and 0.16 DSC for colon cancer (The Medical Segmentation Decathlon). We run nnU-Net's own configuration and give every other arm the same planned patch size, epochs, and augmentation. Beating it means improving held-out-center Hausdorff distance and boundary F-score without losing Dice beyond seed variance.
The strongest flatness competitor is the global Hessian-trace penalty of Learning to Generalize Provably in Learning to Optimize, applied directly to optimizee training. It is the method our claim says is miscalibrated for segmentation, so it must be the best version of itself. That paper reports no segmentation number on Lacuna, so we tune its $\lambda$ over the same grid and trial count as ours, selected on the same validation centers.
Two baselines cover the boundary side. The first is global flatness plus an auxiliary contour loss in the style of Niu et al.. This tests whether simply adding boundary emphasis undoes the damage. The second is GraTa test-time adaptation applied on top of each trained model. GraTa reports a 5.87% average DSC gain over no adaptation and 12.77% on its hardest Domain D. No reported boundary-distance result was found on Lacuna for GraTa or for any of these baselines, so boundary comparisons rest on our own runs. The key ablation is the boundary-only penalty. It isolates the mask itself: if penalizing only the band is no worse than penalizing only the interior, the decomposition is not what drives the effect.
All arms receive matched GPU-hours, not matched epochs. The curvature penalty roughly doubles backward cost, so the no-penalty arm is allowed longer training or seed ensembling at equal compute.
Why not just…?
Why not just use nnU-Net and accept its defaults? For large organs this is the right answer, and we expect no gain there; spleen and liver at 0.94 DSC are effectively solved. The proposal targets the regime where nnU-Net is reported at 0.16 to 0.21 DSC. If the matched-compute nnU-Net arm equals the interior-only arm on held-out-center boundary metrics, the idea fails. The experiment reports that outcome either way.
Why not just add a boundary loss and keep global flatness? This is the most serious threat. A contour loss adds boundary gradient, but it also adds boundary curvature, which the global penalty then flattens. The two pull against each other, so we would expect a tuned compromise and not a recovery. The flatness-plus-contour-loss baseline tests this directly. If it matches the masked penalty, the honest conclusion is that boundary loss weighting is sufficient. The contribution would then shrink to the Stage 1 diagnosis.
Why not just fix shift at test time with GraTa? Test-time adaptation and training-time curvature control are complementary, not exclusive. We therefore apply GraTa on top of every arm. The question is whether the interior-only model remains better after adaptation. If GraTa erases the gap, training-time masking is unnecessary for adapted deployment, though it would still matter wherever per-center adaptation is impractical.
Why not just train at native resolution? Where memory allows, we do, and native resolution is one of the three conditions. The prediction is that the masked penalty's benefit shrinks at native resolution and grows at quarter resolution. A flat benefit across resolutions would argue against our proposed mechanism. The content-adaptive sampler of Marin et al. gives a fourth condition. It tests whether spending pixels on boundaries and relaxing curvature control on them act as substitutes.
Proposed Experiments and Decisive Checks
We use three heterogeneous benchmarks. The first is the five-center optic disc and cup fundus benchmark from GraTa, evaluated leave-one-center-out. The optic cup is a classic low-contrast boundary. The second is multi-center polyp segmentation, training on Kvasir-SEG and CVC-ClinicDB and testing on EndoCV2021, as in Yu et al.. The third is the Decathlon pancreas task, with tumor as the low-contrast target and parenchyma as the higher-contrast control. In the Decathlon we vary resolution but do not claim center shift. BUSI and ISIC2017 serve as secondary single-domain checks of the curvature decomposition. Every result is reported over at least three seeds.
The study rests on three checks. First, the Stage 1 decomposition must show that the boundary band carries a disproportionate share of the Hessian trace, and that the share rises with downsampling and center shift. If it does not, the mechanism is wrong and we stop. Second, at matched compute, the global penalty must improve held-out Dice relative to nnU-Net while degrading Hausdorff distance or boundary F-score. Third, the interior-only penalty must keep most of the Dice gain and remove most of the boundary penalty, with the gap largest at quarter resolution and on the low-contrast targets (cup, pancreas tumor) versus the high-contrast controls (disc, pancreas parenchyma).
A positive result in all three would change how flatness should be specified for dense prediction, including in learned optimizers. Negative results are informative too: a negative second check would mean classification-era flatness transfers to segmentation unchanged.
References
- Learning to Generalize Provably in Learning to Optimize
- $\mu$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- A Medical Image Segmentation Method based on Multi-scale Features and Contour Loss Constrain (S)
- Parallel Res2Net-based Network with ReverseAttention for Polyp Segmentation
- Boundary-Aware Test-Time Adaptation for Zero-Shot Medical Image Segmentation
- nnU-Net: Breaking the Spell on Successful Medical Image Segmentation
- The Medical Segmentation Decathlon
- Gradient Alignment Improves Test-Time Adaptation for Medical Image Segmentation
- Efficient Segmentation: Learning Downsampling Near Semantic Boundaries
- M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptation