Boundary-Gated Band Attention for Few-Label Multispectral Segmentation

Multispectral satellite images record a scene in a dozen or more wavelength bands, but not every band is useful everywhere. Near-infrared may separate vegetation from bare soil inside a field. At the field's edge, what matters is which bands still show a sharp transition after the sensor and the network's downsampling have blurred it. Today's segmentation networks pick one global importance weight per band for the whole image, and they are trained with losses dominated by easy interior pixels. With only a few dozen labeled images, the extra bands can then hurt more than help. The clearest documented case is SwinLS, where adding infrared bands to a Swin Transformer did worse than RGB alone on landslide detection, and the authors fell back to three bands.

We propose to let the boundary loss decide which bands the network trusts, and where. The network predicts a signed distance to the nearest object edge, meaning a map giving each pixel's distance to the boundary, negative inside and positive outside. A boundary loss supervises this map. The predicted distance then switches between two learned band-weighting vectors, one for pixels near an edge and one for interiors. This keeps the useful part of per-pixel spectral attention, namely that band importance varies with location, at the parameter cost of two ordinary channel-attention vectors. That cost is what makes it trainable from 25 to 100 labeled images.

Core Claim

Band importance is spatially non-stationary, and the main axis of that non-stationarity in segmentation is distance to a boundary. If so, a two-state attention gated by a learned distance map should match full per-pixel spectral attention on boundary accuracy. It should also beat global channel attention trained with the same boundary loss. And it should overfit less than either when labels are scarce. The new element is the coupling. In prior work, boundary losses only reweight pixel errors (Bokhovkin, Kervadec et al., BASeg), and spectral attention is learned only from the overall task loss. Here the boundary loss also trains the variable that routes spectral attention.

Mechanism

Let $x(p) \in \mathbb{R}^C$ be the feature vector at pixel $p$ with $C$ channels, and let $\hat d(p)$ be the network's predicted signed distance to the nearest boundary. The rescaled feature is

$$ x'(p) = x(p) \odot \big[\, m(p)\,\alpha_{\text{edge}} + (1 - m(p))\,\alpha_{\text{int}} \,\big], \qquad m(p) = \exp\!\big(-\hat d(p)^2 / \tau^2\big), $$

where $\odot$ is elementwise multiplication, $m(p) \in [0,1]$ is a soft "near-edge" mask, and $\tau$ is a learned bandwidth. The vectors $\alpha_{\text{edge}}, \alpha_{\text{int}} \in [0,1]^C$ are produced by a squeeze-and-excitation style multilayer perceptron, following the global-descriptor design of the attention-based band selection work. The difference is that the input features are pooled separately over the $m$-weighted region and the $(1-m)$-weighted region, rather than over the whole image.

Training uses Dice loss plus a sign-sensitive signed-distance loss on $\hat d$, following Xue et al.. The boundary weight rises over epochs, as in the schedule of Bokhovkin. Because $\tau$ is learned, and can optionally be conditioned on predicted object size, the boundary band width adapts to the data instead of being fixed by hand. A fixed width is a limitation the file notes for buffer-based surrogates.

The gate is inserted at each encoder stage before downsampling. There it can protect edge-relevant bands before striding erases them, which is the failure mode that LIP addresses with learned pooling.

Research Program

Each source plays a different role. RPAN supplies the motivating evidence: its full per-pixel channel-by-height-by-width attention improved spectral reconstruction and noise robustness over channel attention, because spectra change sharply at material boundaries. It also imposes heavy memory cost and has never been tested for segmentation or with few labels. Our gate is a rank-two, boundary-indexed approximation of that tensor. SwinLS constrains the design. Unconstrained mixing of extra bands into spatial attention dilutes focus, so the gate must be able to down-weight bands, and its interior and edge vectors should be inspected for exactly that. CSMNet is the closest prior system. It gates channel, spatial, and global attention on Sentinel-2 farmland but trains with BCE plus Dice, still merges small adjacent parcels, and uses thousands of tiles. We differ in tying the attention to an explicitly supervised distance map and in targeting few-label training.

The negative result of Hwang et al. is a warning. A boundary loss added on top of a representation lacking structural awareness moved AP only from 0.5600 to 0.5620. A boundary loss used only as a pixel reweighting may therefore change little. Our hypothesis is that it matters when it also controls what information enters the features.

We will build the module on a compact U-Net encoder, about ResNet-18 scale, so that the small-sample regime is meaningful. We will then measure accuracy and boundary quality as a function of training set size, and analyze the learned $\alpha_{\text{edge}}$ and $\alpha_{\text{int}}$ against band properties.

Baselines

The naive reference is the same U-Net on RGB only with Dice loss. This is the choice SwinLS effectively made, and the bar that extra bands must clear. A second naive point feeds all bands with Dice loss and no attention, to measure whether the dissipation effect reproduces in a convolutional network at small sample sizes.

The strongest published system on the primary benchmark is SwinLS, which reports 73.99% F1 on the Landslide4Sense test set, against 59.2% for ResUNet. We will rerun its released recipe on our exact training subsets, including its two-stage balanced training and augmentation. Our numbers then reflect data scarcity rather than different splits. For small subsets there is no reported result found on Lacuna for SwinLS or any method on Landslide4Sense at 100 or fewer training patches, so those comparisons rest on our reruns.

On farmland, CSMNet reports 90.85% IoU on the Nong'an County Sentinel-2 data. We will reimplement its attention module in our backbone if the data are obtainable. On Potsdam we will use the infrared-bearing channels and the binary building task, where Bokhovkin reports 93.85% IoU with its hybrid boundary loss.

The decisive ablations isolate the coupling. The first is global squeeze-and-excitation attention trained with the identical boundary loss and schedule. The second is our gate driven by a fixed distance map computed from Sobel edges of the input rather than from the supervised $\hat d$. This tests whether supervision of the gate matters, and echoes the fragility of fixed gradient operators noted by StructSAM. The third is a full RPAN-style per-pixel attention block with the same loss.

Every method gets the same backbone, the same subsets, five seeds per subset size, and an equal hyperparameter search budget of 20 trials on a held-out validation split. Beating the field means two things. At 50 training images, our method should exceed the best of these by at least two points of boundary F1, with non-overlapping seed intervals. Its region IoU should be no worse than one point below any competitor.

Why not just…?

Why not just add a boundary loss to a standard channel-attention network? This is the most serious threat, and it is the first ablation. Bokhovkin shows a boundary surrogate alone gains over Dice, from 92.61% to 93.85% IoU on Potsdam. Much of any improvement could come from the loss rather than the gate. If global attention plus boundary loss matches us within seed noise, the core claim is wrong, and we will report it that way.

Why not just use full per-pixel spectral attention as in RPAN? It is strictly more expressive, and RPAN showed it beats channel attention under 25dB noise, with MRAE 0.15532 against 0.20315. The cost is memory, and many more parameters to fit from few labels. We predict it ties or wins with the full dataset and loses below 100 images. The training-size curve settles this, and if it does not cross, the low-rank gate is only an efficiency trick.

Why not just keep RGB, or compress bands with PCA or a fixed 1×1 convolution? That is what SwinLS did, and it is cheap. But fixed projections apply one weighting everywhere, which is exactly the assumption RPAN's results contradict. The RGB and PCA baselines are run directly. If they match us, the extra bands carry no usable boundary information on these tasks.

Why not just fine-tune a pretrained foundation model with adapters? Hwang et al. found adapter tuning, full fine-tuning, and boundary losses all stuck within 0.002 AP of each other on LIVECell. That suggests that tuning cannot add missing boundary structure. Their setting is single-channel microscopy, though. We will include a frozen DINO-style encoder with adapters on RGB as an additional reference, while acknowledging it can use only three bands without modification.

Proposed Experiments

Training subsets of 25, 50, 100, 400, and all patches are drawn from Landslide4Sense, and correspondingly from Potsdam tiles and, if available, Nong'an tiles. Metrics are region F1 and IoU, boundary F1 at a 2-pixel tolerance, 95th-percentile Hausdorff distance, and a merge rate: the fraction of ground-truth components separated by at most 3 pixels that the prediction joins. That last metric targets the parcel-blending failure CSMNet reports.

The headline result is the curve of boundary F1 against training size for the full method, global attention plus boundary loss, and RPAN-style attention. The claim needs the gap to the second-best method to widen as data shrink.

The interpretive check is falsifiable. If the gate captures something physical, $\alpha_{\text{edge}}$ should favor bands with sharper native resolution and higher local contrast, and $\alpha_{\text{int}}$ should favor spectrally discriminative bands. Swapping the two vectors at test time should hurt boundary F1 far more than region IoU. If the learned vectors are nearly identical, or the swap does nothing, the edge-interior split is not doing the work.

A final stress test adds Gaussian blur to the high-resolution bands only. It asks whether the gate shifts weight accordingly, which separates real boundary-aware routing from a fixed preference learned during training.