PatchRefineNet: Improving Binary Segmentation by Incorporating Signals from Optimal Patch-wise Binarization

Savinay Nagendra, Daniel Kifer

Abstract

Binary segmentation models typically rely on a global threshold to convert probability maps into final masks. However, these models often exhibit "spatial bias"—systematic regional errors where the model is consistently over- or under-confident. We present PatchRefineNet (PRN), a lightweight refinement network designed to mitigate these biases. PRN utilizes a dual-branch architecture and a novel supervision signal derived from optimal patch-wise thresholding. Our experiments demonstrate that PRN consistently improves state-of-the-art base models by 2–3% mIoU across diverse tasks, including satellite road extraction and medical polyp segmentation.

Introduction: The Challenge of Spatial Bias

In binary segmentation, a neural network produces a logit map representing pixel-level probabilities. The standard practice of applying a single global threshold (e.g., 0.5) to this map often fails to account for regional variations. In some areas, the model may be "too optimistic," creating false positives; in others, it may be "too pessimistic," leading to gaps in the predicted mask. This phenomenon, termed spatial bias, is often overlooked by standard pixel-level loss functions. While post-processing methods like Conditional Random Fields (CRFs) can help, they are often computationally heavy and struggle with large-scale systematic errors.

Methodology: PatchRefineNet (PRN)

PRN is designed as a small, plug-and-play U-Net-style network that processes the logit maps of a fixed, pre-trained base model.

Architecture

The network employs a dual-branch strategy to capture multi-scale information:

  1. Global Branch: Analyzes the entire logit map to maintain global consistency and understand long-range spatial relationships.
  2. Local Branch: Divides the logit map into disjoint patches (e.g., 64 x 64 pixels). Each patch is processed independently to identify and correct local biases.
The PatchRefineNet architecture features a dual-branch decoder to process global and local features independently.

Supervision via Optimal Patch-wise Thresholding

The key innovation of PRN lies in its training signal. Instead of relying solely on ground truth, the authors generate pseudo-labels by calculating the optimal threshold for each individual image patch—the threshold that maximizes the Intersection over Union (mIoU) for that specific region. PRN is then trained to predict these "ideal" local adjustments, effectively learning to "nudge" the base model's probabilities toward their optimal local values.

Experimental Results

PRN was evaluated on four diverse datasets: DeepGlobe (road extraction), Kvasir-SEG (polyp segmentation), DUTS (saliency detection), and FSS-1000 (few-shot segmentation).

Quantitative Performance

PRN demonstrated a consistent "rising tide" effect, improving the performance of various base architectures (e.g., U-Net, DeepLabV3+, CoANet) by 2–3% mIoU. For instance, on the DeepGlobe dataset, adding PRN to a CoANet base model increased the mIoU from 67.9% to 70.6%. Notably, PRN maintains constant memory usage regardless of image size, making it more efficient than existing refinement methods like CascadePSP.

Qualitative Improvements

Qualitatively, PRN excels at repairing structural errors. In satellite imagery, it successfully connects fragmented road segments. In medical imaging, it removes "ghost" artifacts and refines the boundaries of detected polyps.

Qualitative comparison showing how PRN corrects spatial biases and refines segmentation masks across road extraction and medical imaging tasks.

Conclusion and Future Work

PatchRefineNet provides a robust, efficient solution for correcting spatial bias in binary segmentation. By leveraging the signal from optimal local thresholding, it allows models to adapt to regional variations that global thresholds ignore. While currently focused on binary tasks, future research could extend this approach to multi-class segmentation by exploring efficient ways to calculate optimal threshold vectors for multiple categories.

Summary updated

Abstract

The purpose of binary segmentation models is to determine which pixels belong to an object of interest (e.g., which pixels in an image are part of roads). The models assign a logit score (i.e., probability) to each pixel and these are converted into predictions by thresholding (i.e., each pixel with logit score $\geq τ$ is predicted to be part of a road). However, a common phenomenon in current and former state-of-the-art segmentation models is spatial bias -- in some patches, the logit scores are consistently biased upwards and in others they are consistently biased downwards. These biases cause false positives and false negatives in the final predictions. In this paper, we propose PatchRefineNet (PRN), a small network that sits on top of a base segmentation model and learns to correct its patch-specific biases. Across a wide variety of base models, PRN consistently helps them improve mIoU by 2-3\%. One of the key ideas behind PRN is the addition of a novel supervision signal during training. Given the logit scores produced by the base segmentation model, each pixel is given a pseudo-label that is obtained by optimally thresholding the logit scores in each image patch. Incorporating these pseudo-labels into the loss function of PRN helps correct systematic biases and reduce false positives/negatives. Although we mainly focus on binary segmentation, we also show how PRN can be extended to saliency detection and few-shot segmentation. We also discuss how the ideas can be extended to multiclass segmentation.
BibTeX
@inproceedings{nagendra2024patchrefinenet,
  author = {Savinay Nagendra and Daniel Kifer},
  title = {{PatchRefineNet: Improving Binary Segmentation by Incorporating Signals from Optimal Patch-wise Binarization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00139},
}
Figure