Abstract
We present ThreshNet, a post-processing method to refine the output of neural networks designed for binary segmentation tasks. ThreshNet uses the confidence map produced by a base network along with global and local patch information to significantly improve the performance of even state-of-the-art methods. Binary segmentation models typically convert confidence maps into predictions by thresholding the confidence scores at 0.5 (or some other fixed number). However, we observe that the best threshold is image-dependent and often even region-specific -- different parts of the image benefit from using different thresholds. Thus ThreshNet takes a trained segmentation model and learns to correct its predictions by using a memory-efficient post-processing architecture that incorporates region-specific thresholds as part of the training mechanism. Our experiments show that ThreshNet consistently improves over current the state-of-the-art methods in binary segmentation and saliency detection, typically by 3 to 5% in mIoU and mBA.
Introduction: The Problem of Spatial Bias
In binary segmentation, a model outputs a "logit map" (probability scores for each pixel), which is typically converted into a final prediction using a global threshold (usually 0.5). However, even the best models suffer from spatial bias: they might consistently over-predict in one region of an image (upward bias) and under-predict in another (downward bias).
For example, in a satellite image, a model might assign a road pixel a low score because of shadows, causing a false negative. Conversely, a bright rooftop might get a high score, causing a false positive. A single global threshold cannot account for these local variations. While one could try to predict a different threshold for every patch, this approach is often too rigid to handle the complex, non-linear errors found in real-world data.
Methodology: PatchRefineNet (PRN)
The authors introduce PatchRefineNet (PRN), also referred to as ThreshNet, a small neural network that sits on top of any "base" segmentation model.
The Architecture
PRN uses a U-Net-style encoder (ResNet-34) and features two distinct decoder branches:
- Global Branch: Processes the entire logit map to capture wide-scale context and relationships between different regions.
- Local Branch: Splits the map into disjoint patches and processes them independently to learn fine-grained, local corrections.
The outputs of these two branches are averaged to produce the final refined logit map. The most novel aspect of PRN is its supervision signal. During training, the authors generate pseudo-labels by finding the optimal threshold for every individual patch in the training set—the threshold that maximizes the Intersection over Union (IoU) for that specific patch. By training PRN to match these "patch-optimized" labels, the network learns to recognize when a base model is systematically "off" in a specific area and automatically shifts those logit scores to correct the bias.
Key Results
PRN demonstrated consistent improvements across diverse tasks including road extraction (DeepGlobe), polyp segmentation (Kvasir-SEG), and saliency detection (DUTS):
- Quantitative Boost: Across various base models (like U-Net and DeepLabV3+), PRN improved mean IoU (mIoU) by 2–3%.
- Boundary Accuracy: The model showed even more significant gains in Mean Boundary Accuracy (mBA), often improving by 4–9%. This suggests PRN is particularly good at cleaning up messy edges and disconnected segments.
- Qualitative Fixes: In road detection, PRN successfully connected road segments that the base model had "broken" due to local downward bias and removed "ghost" roads caused by upward bias.
Conclusion
ThreshNet (PRN) treats a base model's systematic errors as a learnable pattern. By decoupling the refinement process from the base model training, it offers a computationally efficient way to "bolt on" performance gains to existing high-performing systems. Future work aims to extend this region-specific thresholding logic to multi-class segmentation tasks.
Summary updated