Segmentation models label every pixel of an image, yet on large objects they are still sloppy at the edges. Boundary IoU shows how large the effect is. Mask R-CNN trained on COCO scores 66.0 mask AP on large objects but only 25.9 when the score is computed in a thin band around the contour. A long tradition, which this note calls "edge-region duality," treats a boundary and the region it encloses as two views of one object. That tradition built exact guarantees: closed contours, no dangling edge fragments, no pinches. But those guarantees came from linear-program solvers that were only ever tested on a few hundred images. The broader "scalability gap" literature warns that this pattern is common. Structural priors that act as useful regularizers on small clean datasets tend to turn into bottlenecks, or simply stop helping, once data becomes large and varied.
This note proposes a controlled study, paired with a deliberately minimal training objective, to answer one question. When boundary-aware training helps segmentation, is it acting as a regularizer that more data will make unnecessary, or is it correcting a bias in the training objective that more data cannot fix? We think the answer differs by ingredient. The COCO boundary deficit comes mostly from how pixel-summed losses weight large objects, and that weighting does not change with dataset size. Topological and curvature priors, by contrast, behave like classic small-data regularizers. If this is right, practitioners should keep the reweighting and drop the heavy topology machinery at scale. That is the opposite of the obvious "port winding numbers into deep nets" program.
Core Claim
The key step is to notice what modern networks already give for free. If a network outputs one score map per class and each pixel takes the class with the highest score, the resulting label map is a valid partition of the image. The boundaries between labels are closed curves that meet at junctions. No boundary can dangle, because it is defined by the regions on either side. This holds for any number of classes.
This is the multi-class generalization of the zero-crossing argument in From Early Processing to Conceptual Reasoning, where boundaries defined as sign changes of a filtered signal must separate exactly two regions. Region-boundary inconsistency in deep models appears only when a separate edge head is trained alongside the mask head. That is the design used by Graph-Based Region and Boundary Aggregation, which then needs a Boundary Agreement Loss to reconcile the two, and by the Canny-supervised boundary branch in Fourier Boundary Features Network.
So the unsolved problem at scale is not duality of representation. It is where the loss puts its weight. In a pixel-summed cross-entropy, an object of radius $r$ contributes about $r^2$ interior pixels but only about $r$ contour pixels. The contour's share of the gradient therefore falls as $1/r$, and adding images does not change that ratio.
Our claim is testable as a scaling statement. Let $\Delta_B(n)$ be the Boundary AP gain an ingredient gives over plain training when the model sees $n$ training images. We predict two different behaviors:
- For per-object perimeter-normalized weighting, $\Delta_B(n)$ stays roughly flat as $n$ grows from 5% to 100% of COCO.
- For explicit topology and curvature priors, $\Delta_B(n)$ shrinks, and on heterogeneous scenes it may turn negative.
Research Program
We build one objective, which we call the dual-band loss, and make it as simple as possible so that it isolates the reweighting. There is no separate boundary variable. The predicted boundary is always the set of label changes of the highest-scoring class. For each ground-truth instance or segment $k$, we take a band of width $d_k$ around the union of its true contour and its predicted contour. Following Boundary IoU, we use both contours so that the loss catches missed edges as well as spurious ones. Unlike Boundary IoU's fixed 2%-of-diagonal band, $d_k$ scales with the object's own size and is floored at a few pixels. We then normalize so that each object's band contributes in proportion to its perimeter rather than its area:
$$L = L_{CE} + \lambda \sum_k \frac{1}{|B_k|} \sum_{p \in B_k} \ell_{CE}(p) \cdot \frac{|\partial G_k|}{\sum_j |\partial G_j|},$$
where $B_k$ is the set of band pixels for object $k$, $\ell_{CE}(p)$ is the cross-entropy at pixel $p$, $|\partial G_k|$ is the length of object $k$'s ground-truth contour, and $\lambda$ is a single weight tuned on held-out data. Because the band is defined partly by the prediction, it moves as training proceeds. The loss therefore behaves like a coarse active contour without distance transforms. This removes the offline distance-transform computation of Active Boundary Loss and the pretrained alignment network of InverseForm.
Alongside this we implement the structural ingredient the duality literature argues for, in a form cheap enough to run at scale. It is a differentiable topology term that penalizes mismatches in the number of connected components and holes between each predicted and true object, computed on the soft mask within each object's box. We add an optional curvature penalty on the predicted contour. This is a GPU-friendly stand-in for the curvature linear program of Curvature regularity for region-based segmentation, which preserved insect legs on single-object scenes but required about a million variables per image.
We do not claim this stand-in inherits that paper's optimality guarantees. Its role is to represent "explicit structural prior" in the scaling comparison. We also train at least one model with a separate edge head, supervised with an agreement loss, to test directly whether explicit edge variables add anything once boundaries come from the region map.
Baselines
The naive reference is the unmodified model trained with standard losses at each data fraction: Mask R-CNN with a ResNet-50 backbone on COCO and LVIS, and DeepLabV3 on Cityscapes. On COCO its reported full-data numbers are 66.0 Mask $AP_L$ and 25.9 Boundary $AP_L$, from Boundary IoU. We retrain it ourselves at every fraction so that all curves share one codebase and schedule.
The strongest existing competitors are PointRend and Active Boundary Loss. PointRend refines masks by re-predicting uncertain points with a small network, and it attacks exactly the large-object boundary problem. Its reported gains are 0.9 mask AP on COCO, 1.6 AP on high-quality LVIS masks, and 2.8 AP on Cityscapes instances. Boundary IoU reports that PointRend's improvement is 22% in Boundary IoU versus 8% in Mask IoU. No reported absolute Boundary AP for PointRend was found on Lacuna. ABL reports gains of 1.0% mIoU on Cityscapes with DeepLabV3 and 1.22% mIoU on ADE20K with OCR, and it outperforms SegFix on boundary F-score. InverseForm reports 87.0% Cityscapes mIoU with HRNet-48-OCR. We use it as a secondary baseline because its auxiliary network confounds compute comparisons.
Every method gets the same backbone, schedule, data fraction, and an equal hyperparameter search of roughly eight settings per method per fraction, chosen on a held-out split. The dual-band loss counts as beating a baseline if it matches or exceeds that baseline's Boundary AP gain at full data with no extra inference cost. Against PointRend specifically, beating means that the dual-band loss combined with PointRend's head improves on PointRend alone. That would show the two effects are complementary rather than redundant.
The key ablations remove one ingredient at a time:
- Perimeter normalization, replaced by uniform band weighting.
- Scale-adaptive $d_k$, replaced by the fixed 2% band.
- The prediction-side half of the band.
- The topology/curvature term.
- The separate edge head.
Each ablation is run across all data fractions, not only at full data.
Why not just…?
Why not just weight pixels near the ground-truth edge more heavily in cross-entropy? This is the real threat, and our loss is close to it. The differences are that our band also follows the predicted contour, and that each object is normalized by perimeter so large objects stop being drowned out by their interiors. Evidence from InverseForm and ABL that pixel-wise boundary losses are geometrically blind argues that the naive version falls short, but that evidence is not decisive. So naive distance weighting is a baseline and an ablation at every data fraction. If it tracks the dual-band loss, we report that the simple fix suffices. The scaling result about structural priors stands either way.
Why not just use PointRend or higher-resolution masks? PointRend reallocates compute toward boundaries but still trains on pixel-summed losses. PointRend's own paper notes that it cannot repair a missed thin part or a wrongly split region, because it only re-labels uncertain points near an existing boundary. Raising mask resolution grows cost quadratically. Our combination experiment shows whether the loss-side and sampling-side corrections add together.
Why not just trust that more data fixes boundaries? The 40-point gap between mask and boundary AP at full COCO scale argues against it. Still, we do not know how that gap changes with $n$, and the untreated baseline's scaling curve measures exactly that. A shrinking gap would weaken our motivation, and we would report it.
Why not port the exact winding-number or linear-program solvers? The winding-number method takes about 20 seconds per image, handles only figure versus ground, and degrades under occlusion. The label-change representation already gives the consistency those solvers guarantee. What remains is their topological and curvature content, which is exactly the ingredient our scaling test puts under scrutiny.
Proposed Experiments
We train on 5, 10, 25, 50, and 100% of COCO and Cityscapes, subsampling by image, with three seeds per point. We add LVIS for high-quality masks, and use ADE20K as a heterogeneous stress test for the curvature term. Metrics are:
- Mask AP, Boundary AP, and Boundary PQ.
- mIoU with a Boundary IoU variant for the semantic tasks.
- Per-object topology error: component and hole mismatch.
- Recall on thin classes such as poles, bicycles, and riders.
A second axis doubles backbone size, from ResNet-50 to ResNet-101. This separates capacity limits from inductive-bias effects, a protocol gap the scalability literature explicitly names.
Three results would be decisive:
- Scaling curves of the gains. If the dual-band gain in Boundary AP stays within its seed noise from 5% to 100% while the topology/curvature gain declines monotonically, the objective-correction hypothesis holds. If both decline together, it is falsified, and boundary-aware losses are just another small-data regularizer.
- Decomposition by object size. Our account predicts the dual-band gain concentrates on large objects, where the $1/r$ dilution is worst. A uniform gain across sizes would point to a different mechanism.
- The edge-head ablation. If adding an explicit edge head with an agreement loss gives no Boundary AP gain once boundaries come from the label map, that settles whether explicit dual edge variables are worth carrying into large-scale models.
References
- Boundary IoU: Improving Object-Centric Image Segmentation Evaluation
- From Early Processing to Conceptual Reasoning: An Attempt to Fill the Gap
- Graph-Based Region and Boundary Aggregation for Biomedical Image Segmentation
- Fourier Boundary Features Network With Wider Catchers for Glass Segmentation
- Active Boundary Loss for Semantic Segmentation
- InverseForm: A Loss Function for Structured Boundary-Aware Segmentation
- Curvature regularity for region-based image segmentation and inpainting: A linear programming relaxation
- PointRend: Image Segmentation As Rendering
- Winding Number for Region-Boundary Consistent Salient Contour Extraction