Abstract
Binary segmentation models typically rely on a global threshold to convert probability maps into final masks. However, these models often exhibit "spatial bias"—systematic regional errors where the model is consistently over- or under-confident. We present PatchRefineNet (PRN), a lightweight refinement network designed to mitigate these biases. PRN utilizes a dual-branch architecture and a novel supervision signal derived from optimal patch-wise thresholding. Our experiments demonstrate that PRN consistently improves state-of-the-art base models by 2–3% mIoU across diverse tasks, including satellite road extraction and medical polyp segmentation.
Introduction: The Challenge of Spatial Bias
In binary segmentation, a neural network produces a logit map representing pixel-level probabilities. The standard practice of applying a single global threshold (e.g., 0.5) to this map often fails to account for regional variations. In some areas, the model may be "too optimistic," creating false positives; in others, it may be "too pessimistic," leading to gaps in the predicted mask. This phenomenon, termed spatial bias, is often overlooked by standard pixel-level loss functions. While post-processing methods like Conditional Random Fields (CRFs) can help, they are often computationally heavy and struggle with large-scale systematic errors.
Methodology: PatchRefineNet (PRN)
PRN is designed as a small, plug-and-play U-Net-style network that processes the logit maps of a fixed, pre-trained base model.
Architecture
The network employs a dual-branch strategy to capture multi-scale information:
- Global Branch: Analyzes the entire logit map to maintain global consistency and understand long-range spatial relationships.
- Local Branch: Divides the logit map into disjoint patches (e.g., 64 x 64 pixels). Each patch is processed independently to identify and correct local biases.
Supervision via Optimal Patch-wise Thresholding
The key innovation of PRN lies in its training signal. Instead of relying solely on ground truth, the authors generate pseudo-labels by calculating the optimal threshold for each individual image patch—the threshold that maximizes the Intersection over Union (mIoU) for that specific region. PRN is then trained to predict these "ideal" local adjustments, effectively learning to "nudge" the base model's probabilities toward their optimal local values.
Experimental Results
PRN was evaluated on four diverse datasets: DeepGlobe (road extraction), Kvasir-SEG (polyp segmentation), DUTS (saliency detection), and FSS-1000 (few-shot segmentation).
Quantitative Performance
PRN demonstrated a consistent "rising tide" effect, improving the performance of various base architectures (e.g., U-Net, DeepLabV3+, CoANet) by 2–3% mIoU. For instance, on the DeepGlobe dataset, adding PRN to a CoANet base model increased the mIoU from 67.9% to 70.6%. Notably, PRN maintains constant memory usage regardless of image size, making it more efficient than existing refinement methods like CascadePSP.
Qualitative Improvements
Qualitatively, PRN excels at repairing structural errors. In satellite imagery, it successfully connects fragmented road segments. In medical imaging, it removes "ghost" artifacts and refines the boundaries of detected polyps.
Conclusion and Future Work
PatchRefineNet provides a robust, efficient solution for correcting spatial bias in binary segmentation. By leveraging the signal from optimal local thresholding, it allows models to adapt to regional variations that global thresholds ignore. While currently focused on binary tasks, future research could extend this approach to multi-class segmentation by exploring efficient ways to calculate optimal threshold vectors for multiple categories.
Summary updated