Segmentation networks shrink the image as they go deeper so that each unit can see more context. By the time the network makes its prediction, thin object boundaries have already been blurred away. The usual fix is to add a loss that punishes errors near boundaries. That loss acts only at the output, and its signal has to travel back through the same shrinking layers that destroyed the boundary. It also cannot tell a real object edge from a busy texture such as roof tiles, crop rows, or foliage. We propose moving the boundary supervision into the downsampling step itself. Each pooling layer learns a per-pixel "keep this" weight. That weight is trained to be high on true label boundaries and explicitly low on strong image edges that are only texture. The layers that shrink the image then learn to protect object edges and to spend no capacity on texture edges.
The opportunity is that three separately developed ideas already exist, and each misses the piece the others supply. Learnable importance-weighted pooling keeps small targets alive but is trained only end-to-end, with no notion of which details matter. Boundary losses know which details matter but act too late. Texture-versus-shape work shows that edge-sensitive models confuse the two. Our bet is that one auxiliary target on the pooling weights, defined from labels, gives most of the benefit of a boundary loss plus a high-resolution decoder, at lower cost. We also expect it to hold up much better on textured scenes.
Core Claim
The claim has three parts. First, the gain from boundary-weighted losses shrinks as the network's output stride grows, because downsampling discards boundary evidence before the loss can shape it. Second, supervising the pooling importance maps with a semantic-boundary target, while penalizing importance on texture-only edges, recovers that gain. Third, telling texture edges from object edges requires context beyond a 3×3 window, so the module that predicts the importance maps needs a cheap global view even at early, high-resolution stages.
What Prior Work Does and Leaves Open
The method builds on LIP: Local Importance-Based Pooling. LIP predicts a spatial importance logit with a small convolutional module and pools features as an importance-weighted average. It raises ResNet-50 ImageNet top-1 from 76.40% to 78.19% and COCO small-object AP from 21.9% to 24.0%. Its importance maps are learned only through the task loss, and the logit module is purely local. We keep LIP's pooling operator and change what trains it.
A Texture Reconstructive Downsampling for Multi-Scale Object Detection in UAV Remote-Sensing Images takes a competing route. It reconstructs lost high-frequency detail by back-projection, deliberately keeping texture. That is the opposite of what segmentation boundaries need when the texture is irrelevant, so we use it as a contrast.
The boundary-loss papers motivate the problem. Boundary Loss for Remote Sensing Imagery Semantic Segmentation trains a U-Net with a differentiable Boundary-F1 surrogate blended with IoU. BASeg adds a Mahalanobis-angle boundary loss and a state-space global module on a frozen foundation backbone. Both apply supervision only at the output, and neither separates texture transitions from semantic ones. Unmixing Convolutional Features for Crisp Edge Detection (CATS) is the closest in spirit. It names the blur caused by upsampling deep features "feature mixing" and adds a texture-suppression term to its edge loss. Its suppression is still an output-level loss on a purely local convolutional edge detector, and it does not change what the downsampling layers keep.
Decoupling Shape and Texture in SAM-2 via Controlled Texture Replacement (TextureSAM) constrains our design. It shows that texture-sensitivity gains trade off against general mIoU, and that on some data (RWTD) texture boundaries are the semantic boundaries. We handle this by defining "texture-only edge" relative to the labels, never from image statistics alone. If a texture change coincides with a label boundary, it counts as a boundary to keep.
Finally, CCNet supplies the global operator. Criss-cross attention lets each pixel attend along its row and column, at $O(N\sqrt{N})$ cost where $N$ is the number of pixels, and uses 11× less memory than dense non-local blocks. That makes it affordable at stride 4, where CCNet itself never applied it.
Mechanism
At each downsampling stage, LIP-style pooling computes
$$y_p = \frac{\sum_{i \in \Omega_p} e^{s_i} x_i}{\sum_{i \in \Omega_p} e^{s_i}},$$
where $x_i$ is the feature at input pixel $i$, $s_i$ is its predicted importance logit, $\Omega_p$ is the pooling window for output location $p$, and $y_p$ is the pooled feature. We build two label-derived pixel sets at that stage's resolution. $B$ contains pixels within $r$ pixels of a ground-truth class boundary. $T$ contains pixels whose image gradient energy exceeds a percentile threshold but which lie farther than $r$ from any label boundary; these are texture edges. The auxiliary loss is
$$L_{\text{pool}} = -\frac{1}{|B|}\sum_{i\in B} \log \sigma(s_i) \;-\; \frac{\lambda}{|T|}\sum_{i\in T} \log\big(1-\sigma(s_i)\big),$$
where $\sigma$ is the logistic function and $\lambda$ sets how strongly texture edges are suppressed. All other pixels are left unsupervised, so the network can still use importance for small objects as LIP intends. The total objective is the ordinary segmentation loss plus $L_{\text{pool}}$ summed over stages. An output-level boundary loss is optional.
The logit module gets one criss-cross attention layer at stride 4, and runs locally at deeper stages. The reason is concrete: whether a tile seam is a roof edge depends on the extent of the roof, which a 3×3 window cannot see. No hand-tuned smoothing enters the target. This avoids the sensitivity to the smoothing scale that StructureFlow's structure prior suffers from; StructureFlow depends on hand-tuned relative-total-variation smoothing (StructureFlow).
Research Program
The work has three parts. The first is diagnostic. We train the same backbone at output strides 4, 8, 16, and 32, with and without an output-level Boundary-F1 loss. We measure boundary F1 at 1-, 2-, and 3-pixel tolerances, and the norm of the boundary-loss gradient that reaches stride-4 features within $r$ pixels of true boundaries. The hypothesis predicts that the boundary loss's gain falls as stride increases and that the gradient reaching near-boundary shallow features falls with it.
The second part builds boundary-supervised pooling into ResNet-50 and ResNet-101 segmentation models, following LIP's backbone choice so our comparison to LIP is direct. The third part is a texture stress test: we measure whether the texture-negative term, and the global logit module, specifically reduce spurious boundaries inside textured regions.
Baselines
The naive reference is a standard ResNet segmentation model with strided downsampling and cross-entropy or Dice loss. On ISPRS Potsdam, the Boundary Loss paper reports that plain IoU and Dice losses reach 92.56% and 92.61% IoU (Boundary Loss for Remote Sensing Imagery Semantic Segmentation).
The strongest loss-side competitor is that paper's hybrid Boundary-F1/IoU loss, which reports 93.85% IoU on Potsdam. We rerun it on our backbone and schedule rather than copying the number, so architecture differences do not contaminate the comparison. Its reported figure serves as a sanity anchor.
On GCD-25k the strongest reported system is BASeg with its Mahalanobis-angle boundary loss, at 68.86% mIoU (BASeg). Because BASeg relies on a frozen foundation backbone, we run two comparisons. The first adds its boundary loss to our backbone. The second adds boundary-supervised pooling to a trainable adapter in front of BASeg's own pipeline, where feasible.
On Cityscapes and ADE20K the global-context reference is CCNet, which reports 81.9% test mIoU and 45.76% validation mIoU (CCNet). We report validation numbers against our own CCNet reproduction.
On the architecture side we compare against three variants, all with the same output-level boundary loss:
- unsupervised LIP (LIP);
- texture-reconstructive downsampling (TRD);
- CATS-style tracing loss with its texture suppression term (CATS).
The ablations that isolate the key ingredient are:
- boundary-supervised pooling without the texture-negative term ($\lambda=0$);
- the same with a purely local logit module;
- the same auxiliary target applied to a separate edge head instead of the pooling logits.
All runs share data splits, crop size, iteration count, and a fixed hyperparameter search budget per method, including $w$ for the hybrid losses and $\lambda$, $r$ for ours. FLOPs are reported and matched within about 10% by widening baselines where needed.
Beating the strongest baseline means three things together. Ours must match or exceed the rerun hybrid-boundary-loss model on region metrics. It must improve 1-pixel boundary F1 by a margin larger than the seed-to-seed spread across three seeds. And it must reduce texture false boundaries on the stress sets without losing ADE20K mIoU; losing ADE20K mIoU is the failure mode TextureSAM warns about.
Why not just…?
Why not just keep a boundary loss and use a U-Net or feature pyramid with skip connections? This is the most serious threat. Skip connections hand high-resolution features to the decoder, and the output-level boundary loss can then sharpen them. Two pieces of evidence argue against it. LIP's small-object gains appear on backbones that already feed feature pyramids, and CATS attributes remaining blur to exactly this upsample-and-fuse step. Still, neither result tests boundaries under a boundary loss. The stride-sweep diagnostic and a compute-matched U-Net-plus-Boundary-F1 run settle it directly. If that combination matches ours on 1-pixel boundary F1 and on texture false boundaries, the pooling claim fails.
Why not just let LIP learn its importance maps end-to-end? End-to-end importance may already concentrate on boundaries. The ablation of unsupervised LIP plus boundary loss against our supervised version answers this. We also visualize the correlation of learned importance with the $B$ and $T$ sets. Our prediction is that unsupervised LIP puts high importance on texture edges, since those carry high feature energy.
Why not just add an auxiliary edge-prediction head? An edge head also pushes boundary information into shallow features. It does not, however, change what the pooling average keeps. The ablation that places the identical target on a separate head, rather than on the pooling logits, isolates exactly this difference.
Why not just use texture-replacement augmentation? TextureSAM shows augmentation shifts texture sensitivity but costs general mIoU, and it leaves downsampling untouched. We treat augmentation as complementary, and report ours with and without it on the texture stress set.
Why not just run at higher resolution or lower output stride? This buys boundary detail with quadratic activation cost. We report stride-8 models against our stride-16 models at matched FLOPs.
Proposed Experiments
The main benchmarks are ISPRS Potsdam, ISPRS Vaihingen, and GCD-25k for aerial scenes, where roofs, fields, and canopy supply abundant texture edges, and Cityscapes and ADE20K for natural scenes. The texture stress test follows TextureSAM's recipe: pasting textures from the Describable Textures Dataset (DTD) into ADE20K masks, so semantic boundaries are fixed while internal texture varies. We also evaluate on RWTD and STMD. There, label boundaries are texture boundaries, which checks that the label-relative definition does not suppress real edges.
Metrics are mIoU or IoU, boundary F1 at 1-, 2-, and 3-pixel tolerances, and a texture false-boundary rate. That rate is the fraction of predicted class transitions falling in $T$, the set of texture-only edges.
Three decisive checks will settle the claim. The first is the stride sweep: the boundary-loss gain should decay with stride for baselines and stay flatter with boundary-supervised pooling. The second is the texture-edge test: on DTD-replaced ADE20K, the texture-negative term should cut the false-boundary rate relative to $\lambda=0$. Unsupervised LIP should sit at or above the $\lambda=0$ level. The global logit module should help most on large textured objects. The third is the RWTD/STMD control: those scores must not drop relative to the $\lambda=0$ variant. If the first check fails, the premise that downsampling limits boundary losses is wrong. If the second or third fails, texture-aware pooling supervision adds nothing beyond ordinary boundary emphasis.
References
- LIP: Local Importance-Based Pooling
- A Texture Reconstructive Downsampling for Multi-Scale Object Detection in UAV Remote-Sensing Images
- Boundary Loss for Remote Sensing Imagery Semantic Segmentation
- BASeg: Boundary-Aware Remote Sensing Segmentation with Structural Penalties
- Unmixing Convolutional Features for Crisp Edge Detection
- Decoupling Shape and Texture in SAM-2 via Controlled Texture Replacement
- CCNet: Criss-Cross Attention for Semantic Segmentation
- StructureFlow: Image Inpainting via Structure-aware Appearance Flow