Residual-Guided Point Boosting for Thin-Structure Segmentation

Segmentation networks shrink images internally to capture context. The shrinking erases thin, high-detail structures: retinal capillaries, poles, chair legs. Refinement methods try to repair this, but they have a blind spot. They revisit only the pixels where the network is visibly unsure. A thin vessel that the coarse model confidently labels as background is never revisited, because the model is not unsure about it. It is just wrong. We propose to direct refinement at the model's errors rather than its uncertainty. The method is a short chain of small correction stages, built the way gradient boosting builds a model: each new stage is trained on the mistakes left by the stages before it. Each stage has two parts. One predicts where the current model's error is large. The other corrects the prediction only at those locations, by adding a small change to the output scores.

In two sentences for a colleague: boosting says the next learner should fit what the current ensemble gets wrong, and PointRend says refinement should spend compute only on a sparse set of points. We combine them. We train a sequence of point-level correctors, each fitted to the loss gradient left by the frozen previous stages, and each paired with a learned map that predicts where that gradient is large. That map lets the chain find confidently wrong thin structures that uncertainty-based selection never visits.

Core Claim

PointRend is the method closest to this proposal. It starts from a coarse mask, repeatedly picks the points whose class probability is nearest 0.5, and reclassifies them with one lightweight shared network. The benchmark gains are clear: 1.2% mIoU over DeepLabV3 on Cityscapes and 1.6 AP against the high-quality LVIS masks. The PointRend paper itself also states the limitation this proposal targets. If the coarse stage omits a structure or breaks its connectivity, refinement cannot recover it. The reason is that a probability near 0.5 is only a stand-in for "wrong." Gradient boosting, as laid out in Gradient boosting machines, a tutorial, instead sends capacity to wherever the loss gradient is large, whether or not the model is confident there.

We claim three things. First, a predicted residual map finds missed thin structures that probability-near-0.5 selection skips. The map is trained to estimate the magnitude of the per-pixel loss gradient, and it is conditioned on image features, not only on the model's output probabilities. Second, freezing earlier stages makes this map learnable. With the earlier stages fixed, the residual it must predict stays the same throughout its training, instead of shifting every time the model it is diagnosing updates. Third, the correction should be an additive update with a small step size (shrinkage), not an overwrite. This keeps later stages from undoing context that is already correct. That undoing is the "overshoot" PIDNet describes, where strong semantic features wash out delicate edges.

Concretely, let $F_{m-1}(p)$ be the class scores at pixel location $p$ after $m-1$ stages, and let $L$ be the per-pixel loss, either cross-entropy or a boundary-weighted variant. Stage $m$ uses the pseudo-residual, which is the negative gradient of the loss with respect to the current scores:

$$r_m(p) = -\frac{\partial L(y_p, F_{m-1}(p))}{\partial F_{m-1}(p)}.$$

Stage $m$ has two trainable parts. The first is an error locator $e_m$, trained to regress $\lVert r_m(p)\rVert$ densely at low resolution. The second is a point corrector $h_m$, a small network over bilinearly sampled multi-scale features, fitted to $r_m$ at the top-$k$ locations ranked by $e_m$. The update is

$$F_m(p) = F_{m-1}(p) + \lambda\, h_m(p)\quad\text{for the selected } p,$$

where $\lambda$ is the step size (shrinkage factor) and scores at unselected points are left unchanged.

Existing work on this intersection differs in specific ways:

  • Error Correction for Dense Semantic Image Labeling uses a single correction network with warping, replacement, and fusion branches. It is dense, it has one stage, and it does not fit residuals. We use several stages, a sparse set of points, and loss-gradient targets.
  • Correction Learning for Medical Image Segmentation corrects hand-picked error types with superpixels and rules. We learn where errors are.
  • ResFGB and ResFGB-FW supply the stagewise functional-gradient recipe. They applied it only to global-vector classification on MNIST-scale data, not to dense spatial outputs.

Research Program

We build the chain on top of a frozen, pretrained coarse segmenter. For natural scenes this is DeepLabV3, matching PointRend's semantic setting. For vessels it is a standard U-Net. The chain has $M$ stages, with $M$ between 1 and 6. Each stage's error locator is a two-layer convolutional head on backbone features plus the current output scores. Each stage's point corrector is a PointRend-sized network.

Stages are trained one at a time. Stage $m$ is trained with all earlier stages frozen, on a training split. The step size $\lambda$ and the point budget $k$ are chosen on validation data.

Two parts of the design need care.

The first is selection coverage at test time, when no ground truth exists. During training we select points by predicted residual, true residual, and a random sample mixed together. This follows PointRend's importance-sampling practice, adapted so the corrector sees the locator's actual picks.

The second is overfitting in later stages. The file notes that stagewise neural boosting has generalization bounds that grow badly with depth unless training data are split across stages. We therefore test two variants: stages trained on disjoint folds of the training set, and stages that share the data but use strong shrinkage.

Beyond accuracy, we measure how well each selection rule recovers true errors. For a fixed point budget, we compute recall of the true top-residual pixels under three rules: probability-near-0.5, the predicted residual map, and random selection. We report this separately for confident errors (predicted probability above 0.9 for the wrong class) and unconfident ones. This is the mechanism the claim depends on, so it is checked directly rather than inferred from mIoU.

Baselines

The naive reference is the frozen coarse segmenter with bilinear upsampling. It shows how much of the gap to ground truth is recoverable at all.

The strongest competitor is PointRend, run on the same frozen backbone. It gets the same number of refinement points per image, the same number of subdivision steps, a corrector of the same size, and the same hyperparameter search budget. PointRend is the strongest competitor because it already solves the compute side of the problem. The file reports it at 1.2% mIoU over DeepLabV3 on Cityscapes, 0.9 AP over Mask R-CNN on COCO, and 1.6 AP on LVIS masks. It gives no absolute Cityscapes mIoU in this setting, so we will reproduce PointRend ourselves and compare against our reproduction. Beating it means a statistically reliable gain on thin-class IoU and boundary F-score at an equal point budget, while overall mIoU does not drop.

The second competitor is dense learned correction from Error Correction for Dense Semantic Image Labeling. On PASCAL VOC 2012 it raised DeepLabv2 from 79.1% to 80.4% mIoU in 396ms per image, versus 79.7% and 2386ms for dense CRF post-processing. We reimplement it on the same backbone and report both accuracy and runtime.

On thin-structure medical data, the reference point is Study Group Learning, which reports an AUC of 0.9857 on DRIVE. That model was trained in a noisy-label setting, so we report it as a reference and do not claim a head-to-head comparison unless our protocol matches theirs.

Four ablations isolate the ingredients:

  • Selection-only ablation: PointRend with its uncertainty rule replaced by our residual locator, keeping its single shared head. This isolates whether the selection rule matters.
  • Shared-head ablation: all stages share one corrector, isolating whether separate stagewise learners matter.
  • Joint-training ablation: all stages trained together, isolating whether freezing matters.
  • Oracle-selection ablation: points chosen by the true residual. This gives an upper bound on what any selection rule could achieve.

Why not just…?

Why not just swap PointRend's selection rule for a learned error predictor and stop there? This is the most serious threat, and the selection-only ablation tests it directly. If it matches the full chain, the honest conclusion is that residual-guided selection is the contribution and stagewise boosting is unnecessary, and we would report it that way. We expect a gap for one reason. A single shared head trained end-to-end sees a mix of easy and hard points that keeps shifting. A separate stage fitted to a frozen ensemble's residual gets a target that stays fixed during its training, and that is what the tutorial on boosting credits for the stability of small shrunken steps.

Why not just train a cascade of refinement stages jointly with supervision at every stage? Joint training is simpler and often works. But the error locator's target is the residual of the model it diagnoses. If that model changes every step, the locator is chasing a moving target. The joint-training ablation measures this. If joint training matches frozen stagewise training, we drop the freezing.

Why not just use a U-Net with skip connections and a boundary-aware loss? Skip connections do carry detail past the bottleneck. However, Gated-SCNN shows that strong end-to-end models still blur thin classes: its separate shape stream added about 7% IoU on poles and traffic signs and 4% boundary F-score. PIDNet attributes the remaining failures to context overshoot. We will train the same U-Net with our boundary-weighted loss as an extra baseline. That comparison separates the effect of the loss from the effect of the chain.

Why not just apply a dense CRF? On PASCAL VOC, the error-correction paper already found dense CRF refinement both weaker (79.7% versus 80.4% mIoU) and six times slower than a learned corrector. A CRF also cannot restore a structure that the coarse model never predicted, because it only smooths labels according to image similarity.

Proposed Experiments

On Cityscapes, with a frozen DeepLabV3, we compare the naive upsampled baseline, PointRend, dense error correction, the four ablations, and the full chain. We report mIoU, per-class IoU for thin classes (poles, traffic signs, riders), and boundary F-score within a narrow band around object edges. We also report IoU as a function of distance from boundaries, following the distance-based evaluation Gated-SCNN used.

On DRIVE and CHASE_DB1, with a frozen U-Net, we report AUC and a centerline-connectivity measure for vessels thinner than three pixels. These are exactly the confident false negatives the claim concerns.

On PASCAL VOC 2012, we rerun the comparison against learned error correction and dense CRF with runtimes, to connect to the published 79.1/79.7/80.4 numbers.

Three checks decide the question.

  1. Mechanism check. At equal point budget, the predicted residual map must recall substantially more confident errors than probability-near-0.5 selection, and closing that gap must translate into thin-class IoU gains. If recall improves but IoU does not, the corrector, not selection, is the bottleneck.
  2. Stages check. Performance should rise over the first few stages and then level off, with the step size $\lambda$ controlling the trade-off. If stage 2 and beyond add nothing over stage 1, the "chain" is really a single corrector, and we report the method as residual-guided point refinement.
  3. Over-correction check. Using the oracle-selection ablation, we test whether additive shrunken updates damage correct interior pixels less than overwriting updates do.