Each Image Fits Its Own Boundary: Closed-Form Per-Image Classifiers for Low-Contrast Lesion Segmentation

In colonoscopy, ultrasound and MRI, the hardest part of segmenting a lesion is its edge. Flat polyps and diffuse tumors fade into surrounding tissue, and the network's prediction at the border is close to a coin flip. The difficulty grows when the test images come from a different hospital, with different lighting and tissue color. Yet almost every image contains something useful: a region the network is sure is lesion and a region it is sure is background. This proposal fits a tiny linear classifier, separately for every image, on the network's own confident pixels. The classifier is solved exactly in one step by regularized least squares and then applied to the uncertain border pixels. It is cheap, needs no gradient steps at test time, and can be trained end-to-end. The decision boundary between lesion and tissue is therefore recomputed from each image's own appearance rather than fixed by the training hospitals.

The tool comes from an older literature on "collaborative representation", which showed that closed-form ridge regression (least squares with a squared-norm penalty) classifies as well as iterative sparse coding at a tiny fraction of the cost. Later work showed that such solvers can sit inside a deep network as a layer that gradients pass through. Here the solver is the core mechanism. Low-contrast boundaries are the application. Attention modules decide which pixels count as trustworthy examples and which feature channels define the space the solver works in. The weight that blends the solver's output with the ordinary decoder is read off the solver's own uncertainty instead of being learned by a separate gating network.

Core Claim

A differentiable, confidence-weighted ridge regression, solved per image on the network's own confident interior and background pixels, resolves ambiguous boundary pixels better than attention-based interior-to-boundary refinement. The advantage should be largest under cross-dataset shift, and the cost is one small matrix inversion per image. Let $F \in \mathbb{R}^{d \times n}$ be decoder features at the $n$ pixels of a 1/8-resolution map after channel reweighting. Let $W = \mathrm{diag}(w)$ hold per-pixel support weights, let $t \in [0,1]^n$ be soft labels from the coarse prediction, and let $\lambda > 0$ be a learned ridge strength. The per-image classifier and its uncertainty at a pixel feature $x$ are

$$\theta = (F W F^\top + \lambda I_d)^{-1} F W t, \qquad s(x) = x^\top (F W F^\top + \lambda I_d)^{-1} x .$$

Here $s(x)$ is, up to scale, the posterior variance of Bayesian linear regression. It is large when $x$ resembles none of the trusted pixels. The final logit is $\alpha(x)\,\theta^\top x + (1-\alpha(x))\,z(x)$, where $z(x)$ is the decoder logit and $\alpha(x) = \exp(-s(x)/\tau)$ with one learned temperature $\tau$. With $d \approx 64$ channels, the inversion is a $64 \times 64$ system per image, whatever the image size.

The new element is not ridge regression in a segmenter. Meta Learning with Differentiable Closed-form Solver for Fast Video Object Segmentation already maps features to masks in closed form. That work, however, learns from a separately annotated reference frame and applies the map to later frames. Here there is no annotation at test time. The support set is the same image's own confident pixels, the support weights replace hand-set boundary bands, and the solver's variance controls where its answer is trusted.

Positioning Against Prior Work

Two papers motivate the gap. Polyper: Boundary Sensitive Polyp Segmentation is closest in spirit: it uses confident interior features to disambiguate the blurry rim through cross-attention. Polyper finds the rim by eroding and dilating the coarse mask with fixed kernels, and its ablation shows a 4.40-point mIoU drop when this interior/boundary separation is removed. This is evidence that interior-to-boundary transfer matters. It also leaves two weaknesses: the band width is static, and softmax cross-attention weights each interior atom independently. Multi-scale Information Sharing and Selection Network with Boundary Attention for Polyp Segmentation (MISNet) shows the stakes under shift. Its boundary attention and learned balancing gates limit the drop on unseen ETIS images to 16.8%, against more than 38% for competitors, but the gates are ordinary convolutions with no link to how reliable the evidence is.

Three papers supply the method. Sparse representation or collaborative representation: Which helps face recognition? argues that the dictionary's joint, collaborative reconstruction carries the discriminative power, not sparsity. It reports a 1,636× speedup with matching accuracy, which licenses an $\ell_2$ solver over sparse coding. Regularized Robust Coding for Face Recognition shows that reweighting samples by reliability rescues least squares under heavy corruption (67.1% vs 7.1% for sparse coding at 90% corruption). Our weight matrix $W$ plays that role, except that it comes from attention and confidence rather than iterated residuals, so the solve stays one-step. Meta-learning with differentiable closed-form solvers (R2-D2) supplies backpropagation through the solve and the Woodbury identity for moving between feature and sample dimensions. It also reports a 12–14% accuracy drop when the feature extractor is frozen, which is why we train end-to-end rather than bolt the solver onto a pretrained segmenter.

Attention work constrains the design. CBAM: Convolutional Block Attention Module shows that combining average and max pooling beats average-only channel attention, with ImageNet ResNet-50 top-1 error of 22.66% vs 23.14% for squeeze-and-excitation. We reuse its dual-pooling channel and spatial modules to produce the channel reweighting and the spatial term of $w$. Attentional Feature Fusion names the "initial integration bottleneck": a learned gate needs a preliminary blend of the two streams before it can decide how to blend them. Our gate $\alpha$ avoids this because it depends only on the solver's variance, not on a mixture of the streams. Multi-Level Network Embedding with Boosted Low-Rank Matrix Approximation critiques uniform static fusion, which motivates a spatially varying $\alpha$ at all. Boundary-enhanced local-global collaborative network for medical image segmentation (BELGNet) uses "collaborative" only to mean two network branches, not closed-form representation, so it is a performance reference rather than a methodological neighbor.

Research Program

The base network is a Res2Net-50 encoder with a partial decoder and reverse attention in the style of Parallel Res2Net-based Network with ReverseAttention for Polyp Segmentation. It produces a coarse lesion map $p$ and decoder logits $z$.

A CBAM-style channel module turns the 1/8-scale decoder features into the reweighted $F$. The support weight is $w_j = |2p_j - 1| \cdot a_j$, where $a_j$ is a CBAM spatial attention value. Rim pixels the decoder is unsure about therefore get small weight automatically, and the effective boundary band adapts to lesion size and shape without morphological kernels. The soft targets $t$ are the coarse probabilities $p$, with a stop-gradient so the solver cannot simply copy the decoder.

The solve, the variance, and the fused logit are computed per image and upsampled. A refinement path at 1/4 scale is tried only if 1/8 proves too coarse for thin rims. Training uses the standard Dice plus weighted cross-entropy loss on the fused output, plus a small auxiliary loss on the closed-form output alone. The auxiliary loss keeps the solver branch from collapsing to $\alpha \to 0$, the degeneracy that Attentional Feature Fusion-style gates are prone to.

Analysis has three parts. The first is where $\alpha$ is high: it should be high on rims and low on missed or out-of-distribution regions. The second is whether $s(x)$ predicts error in the boundary band. The third is how the gain varies with the regularization strength $\lambda$.

A known limitation, shared with Polyper and reverse attention, is that a lesion the coarse map misses entirely provides no interior support. The design response is that such pixels get high $s(x)$ and fall back to the decoder. The method claims nothing about recovering detections, and we will report missed-lesion rates separately so that boundary gains are not confused with detection gains.

Baselines

The naive reference is the base network trained identically with no solver branch. Any gain must appear relative to it, both in overall Dice and inside a boundary band defined from ground truth as pixels within a fixed distance of the true contour. The band is used for evaluation only and never at inference.

The strongest existing method on the hardest benchmark is Polyper, which reports an mDice of 86.51% on ETIS against 81.00% for SegT (Polyper: Boundary Sensitive Polyp Segmentation). The file does not state Polyper's training protocol or backbone, so we will retrain it with its public settings under our exact protocol and quote its own number alongside. We will also report CaraNet's 74.7% ETIS mDice (CaraNet: Context Axial Reverse Attention Network for Segmentation of Small Medical Objects) and MISNet's 0.903 Kvasir-SEG and 0.918 CVC-ClinicDB mDice (Multi-scale Information Sharing and Selection Network with Boundary Attention for Polyp Segmentation). All retrained methods share the train/test splits, input resolution, augmentation, epoch budget, and a matched hyperparameter search of equal size. Beating the strongest competitor means a higher mean ETIS mDice over three seeds, with non-overlapping seed ranges, and a larger margin inside the boundary band than outside it.

The key ablation isolates the closed-form solve. We keep the identical support set and weights $w$ but replace ridge regression with softmax cross-attention from uncertain pixels to the weighted support, which is the Polyper-style operator, at matched parameters and FLOPs. A second ablation replaces ridge with nearest-prototype matching: compare each pixel to the weighted lesion mean and background mean. Further ablations swap the variance gate for an AFF-style learned convolutional gate, swap the adaptive weights $w$ for a fixed eroded/dilated band, and run the solver with a frozen backbone to test the R2-D2 finding in this setting.

Why not just…?

Why not just use Polyper's cross-attention, which already lets the interior inform the boundary? This is the main threat. Softmax attention is a kernel-weighted average in which each support pixel votes independently. Ridge regression accounts for correlations among support pixels and fits a decision direction, which is the collaborative effect that Sparse representation or collaborative representation: Which helps face recognition? credits for its accuracy. Whether that matters for lesion rims is an empirical question. The matched-support attention ablation settles it. If attention ties ridge, the closed-form claim is dropped and only the adaptive support weighting survives as a contribution.

Why not just compare uncertain pixels to the mean lesion and background feature? Prototype matching is cheaper still and also adapts per image. It ignores within-image variance, though: specular highlights and folds make "background" multimodal, and a single mean misplaces the decision boundary. Ridge regression with whitening by $FWF^\top$ handles this, but the claim stands or falls on the prototype ablation, especially on ETIS.

Why not just fine-tune the network on each test image's confident pixels with a few gradient steps? Test-time fine-tuning targets the same per-image adaptation. It is slow, sensitive to step size, and hard to train through. Meta-learning with differentiable closed-form solvers reports 1 minute 23 seconds for 10,000 episodes versus over 6 minutes for MAML at competitive accuracy. We will run a test-time-SGD baseline that updates only a linear head on the same pseudo-labels with a tuned step budget, and report accuracy against wall-clock time. If it matches accuracy, the remaining argument is speed and the ability to meta-train, and we will state the conclusion in those terms.

Why not just use a bigger backbone or more decoder stages? Attentional Feature Fusion shows that better fusion beats depth on ImageNet, with iAFF-ResNet-50 at 20.2% vs ResNet-101 at 23.2% top-1 error. That is classification, not boundaries, so we add a capacity-matched control: extra decoder stages whose FLOPs equal the solver branch.

Proposed Experiments

Training follows the common polyp protocol: train on Kvasir-SEG and CVC-ClinicDB, test in-distribution on their held-out splits and out-of-distribution on ETIS. BUS-B breast ultrasound serves as a speckle-noise domain, with BELGNet reporting 89.80% DSC (Boundary-enhanced local-global collaborative network for medical image segmentation). The BraTS 2018 necrotic-core 2D subset, where CaraNet reports 63.1% mDice, serves as an MRI domain. Metrics are mDice and mIoU, Dice and boundary F-score inside the ground-truth boundary band, calibration error in the band, missed-lesion rate, and per-image latency.

The first decisive check is the shift prediction. If per-image fitting is the right idea, the gain over the naive network and over the attention ablation should be clearly larger on ETIS than on the Kvasir/ClinicDB test splits, and concentrated in the boundary band. A uniform small gain everywhere would mean the branch is just extra capacity.

The second check is whether the solver variance $s(x)$ ranks band pixels by error better than the decoder's own confidence $|2p-1|$, measured by area under the risk-coverage curve. If it does not, the variance gate has no justification and the learned gate is used instead.

The third check is cost. The solver branch should add under a few milliseconds per image at 352×352 input. Combined with the matched-support ablation, these three checks decide whether the contribution is "closed-form per-image boundary classifiers" or the more modest "confidence-weighted interior support".