Abstract
The purpose of binary segmentation models is to determine which pixels belong to an object of interest (e.g., which pixels in an image are part of roads). The models assign a logit score (i.e., probability) to each pixel and these are converted into predictions by thresholding (i.e., each pixel with logit score $\geq \tau$ is predicted to be part of a road). However, a common phenomenon in current and former state-of-the-art segmentation models is spatial bias—in some patches, the logit scores are consistently biased upwards and in others they are consistently biased downwards. These biases cause false positives and false negatives in the final predictions. In this paper, we propose PatchRefineNet (PRN), a small network that sits on top of a base segmentation model and learns to correct its patch-specific biases. Across a wide variety of base models, PRN consistently helps them improve mIoU by 2-3%. One of the key ideas behind PRN is the addition of a novel supervision signal during training. Given the logit scores produced by the base segmentation model, each pixel is given a pseudo-label that is obtained by optimally thresholding the logit scores in each image patch. Incorporating these pseudo-labels into the loss function of PRN helps correct systematic biases and reduce false positives/negatives. Although we mainly focus on binary segmentation, we also show how PRN can be extended to saliency detection and few-shot segmentation. We also discuss how the ideas can be extended to multiclass segmentation.
Introduction: The Problem of Spatial Bias
In binary segmentation, a neural network typically assigns a "logit score" to every pixel, representing the probability of that pixel belonging to the target class. To generate a final prediction, a global threshold (often 0.5) is applied. However, even state-of-the-art models suffer from spatial bias. In certain regions of an image, the model may be systematically "too shy" (assigning low scores to actual objects) or "too bold" (assigning high scores to background noise).
Standard training often fails to address this because pixel-level loss functions do not explicitly encourage the network to recognize regional trends. If an entire patch of road is assigned a logit of 0.2, a standard model may never learn to "lift" that specific region's scores to meet a global threshold.
Methodology: PatchRefineNet (PRN)
PatchRefineNet is a lightweight, "plug-and-play" post-processing network designed to sit on top of any base segmentation model. It takes the base model's logit map as input and outputs a refined, de-biased map.
Dual-Branch Architecture
PRN employs a dual-branch U-Net-style architecture to handle both global context and local refinement:
- Global Branch: Processes the entire logit map to understand large-scale relationships across the image.
- Local Branch: Splits the image into $k$ disjoint patches and processes them independently. This allows the network to focus specifically on the unique local biases present within each patch.
Novel Supervision via Pseudo-Labels
The core innovation of PRN is its training signal. During training, the authors calculate pseudo-labels by identifying the optimal threshold for each individual image patch—the specific value that maximizes the Intersection over Union (mIoU) for that patch. By training PRN to predict these optimally thresholded maps, the network learns to recognize when a patch is generally biased. The final loss function combines standard ground truth (for fine details) with these pseudo-labels (for regional bias correction).
Experimental Results
The researchers evaluated PRN across diverse datasets, including DeepGlobe (satellite road extraction), Kvasir-SEG (medical polyp segmentation), DUTS (saliency detection), and FSS-1000 (few-shot segmentation). PRN provided a consistent performance lift across all tasks:
- DeepGlobe: Improved the CoANet base model from 67.9% to 70.6% mIoU.
- Kvasir-SEG: Boosted the SSFormer-S model from 86.8% to 89.1% mIoU.
- Saliency Detection: On the DUTS dataset, PRN improved mIoU by +3.8% and Mean Boundary Accuracy (mBA) by +7.4%.
PRN is model-agnostic and computationally efficient. Since it is trained on the fixed outputs of a base model, there is no need to retrain massive backbone architectures from scratch.
Limitations and Future Work
PRN currently requires training on the validation set of the base model to learn the specific types of errors the model makes on unseen data. Additionally, while the paper outlines a theoretical framework for multi-class segmentation, finding the "optimal threshold" in multi-class scenarios becomes a high-dimensional search problem that is currently computationally expensive. Future research will focus on scaling this patch-wise binarization logic to more complex, multi-category scenes.
Summary updated